Packet coding based network communication

ABSTRACT

A method for data communication between a first node and a second node includes forming one or more redundancy messages from data messages at the first node using an error correcting code and transmitting first messages from the first node to the second node over a data path, the transmitted first messages including the data messages and the one or more redundancy messages. Second messages are received at the first node from the second node, which are indicative of: (i) a rate of arrival at the second node of the first messages, and (ii) successful and unsuccessful delivery of the first messages. A transmission rate limit and a window size are maintained according to the received second messages. Transmission of additional messages from the first node to the second node is limited according to the maintained transmission rate limit and window size.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.16/780,275, filed Feb. 3, 2020, which is a continuation-in-part of U.S.patent application Ser. No. 15/061,267, filed Mar. 4, 2016, now U.S.Pat. No. 10,554,565, which claims the benefit of U.S. ProvisionalApplication No. 62/189,509, filed Jul. 7, 2015.

U.S. patent application Ser. No. 16/780,275, filed Feb. 3, 2020, is alsoa continuation-in-part of U.S. patent application Ser. No. 16/277,055,filed Feb. 15, 2019, which is a continuation of U.S. patent applicationSer. No. 15/818,171, filed Nov. 20, 2017, now U.S. Pat. No. 10,333,651,which is a continuation of U.S. patent application Ser. No. 14/935,966,now U.S. Pat. No. 9,825,733, filed on Nov. 9, 2015, which claims thebenefit of U.S. Provisional Application No. 62/076,595, filed Nov. 7,2014.

U.S. patent application Ser. No. 16/780,275, filed Feb. 3, 2020, is alsoa continuation-in-part of U.S. application Ser. No. 16/456,471, filedJun. 28, 2019, which is a continuation of U.S. application Ser. No.15/972,767, filed May 7, 2018, now U.S. Pat. No. 10,425,306, which is acontinuation of U.S. application Ser. No. 14/935,885, filed Nov. 9,2015, now U.S. Pat. No. 9,992,088, which claims the benefit of U.S.Provisional Application No. 62/076,583, filed Nov. 7, 2014.

U.S. patent application Ser. No. 16/780,275, filed Feb. 3, 2020, is alsoa continuation-in-part of U.S. application Ser. No. 16/176,718, filedOct. 31, 2018, which is a continuation of U.S. patent application Ser.No. 14/936,010, filed Nov. 9, 2015, now U.S. Pat. No. 10,320,526, whichclaims the benefit of U.S. Provisional Application No. 62/076,612, filedNov. 7, 2014.

U.S. patent application Ser. No. 16/780,275, filed Feb. 3, 2020, is alsoa continuation-in-part of U.S. patent application Ser. No. 16/456,543,filed Jun. 28, 2019, which is a continuation of U.S. application Ser.No. 15/972,800, filed May 7, 2018, which is a continuation of U.S.application Ser. No. 15/061,211, filed Mar. 4, 2016, now U.S. Pat. No.9,979,664, which claims the benefit of U.S. Provisional Application No.62/189,509, filed Jul. 7, 2015.

U.S. patent application Ser. No. 16/780,275, filed Feb. 3, 2020, is alsoa continuation-in-part of U.S. application Ser. No. 15/972,849, filed,May 7, 2018, which is a continuation of U.S. application Ser. No.14/936,049, filed Nov. 9, 2015, now U.S. Pat. No. 9,992,126, whichclaims the benefit of U.S. Provisional Application No. 62/076,709, filedNov. 7, 2014.

U.S. patent application Ser. No. 16/780,275, filed Feb. 3, 2020, is alsoa continuation-in-part of U.S. application Ser. No. 15/972,898, filedMay 7, 2018, which is a continuation of U.S. application Ser. No.15/061,043, filed Mar. 4, 2016, now U.S. Pat. No. 9,992,128, whichclaims the benefit of U.S. Provisional Application No. 62/189,509, filedJul. 7, 2015.

U.S. patent application Ser. No. 16/780,275, filed Feb. 3, 2020, is alsoa continuation-in-part of U.S. application Ser. No. 16/733,921, filedJan. 3, 2020, which is a continuation of U.S. application Ser. No.15/060,877, filed Mar. 4, 2016, now U.S. Pat. No. 10,530,700, whichclaims the benefit of U.S. Provisional Application No. 62/189,509, filedJul. 7, 2015.

U.S. patent application Ser. No. 16/780,275, filed Feb. 3, 2020, is alsoa continuation-in-part of U.S. application Ser. No. 16/164,022, filedOct. 18, 2018, which is a continuation of U.S. application Ser. No.15/060,908, filed Mar. 4, 2016, now U.S. Pat. No. 10,135,746, whichclaims the benefit of U.S. Provisional Application No. 62/189,509, filedJul. 7, 2015.

U.S. patent application Ser. No. 16/780,275, filed Feb. 3, 2020, is alsoa continuation-in-part of U.S. application Ser. No. 16/165,041, filedOct. 19, 2018, which is a continuation of U.S. application Ser. No.15/060,925, filed Mar. 4, 2016, now U.S. Pat. No. 10,129,159, whichclaims the benefit of U.S. Provisional Application No. 62/189,509, filedJul. 7, 2015.

The entire disclosures of each of the above applications areincorporated herein by reference.

BACKGROUND

This document relates to protocols for communicating over data networks,and more specifically, in at least some examples, to the use of packetcoding based protocols for communication over packet switched networks,for instance, over the Internet.

Data communication has benefited from the near-universal use of theInternet Protocol (IP) on the interconnection of networks that form theInternet. The endpoints of communication connections or sessions set-upover the Internet may include servers, which may be in data centersco-located on “backbones” of the Internet, user devices on wired orwireless local area networks, and mobile devices on various generationsof cellular telephone technology (e.g. 3G, 4G, LTE). Local area networksmay be coupled to high-speed backbones of the Internet via facilities ofInternet Service Providers (ISPs), with “last mile” technologies rangingfrom digital subscriber loop (DSL) to hybrid-fiber coax to all-opticalnetworks. In some cases, networks may include satellite communicationlinks which may have very different delay characteristics than, forexample, terrestrial optical networks.

The communications paths that data packets follow in travelling fromwhere they originate to their destination(s) may typically traversemultiple different types of links and/or networks. Each link and/ornetwork may be supported by operating equipment such as servers,buffers, transmission links and the like, and may be characterized byparameters such as capacity, congestion, delay, packet loss, timing ofdata transfer and the like. Furthermore, transition points, alsosometimes referred to as “peering points” between types of networks mayimpose some restrictions on how data may flow through the networks.

In addition to characteristics that are inherent in the network designs,policy characteristics imposed by network operators may affect howtraffic flows across networks. For example, certain types of trafficand/or connections may be prioritized and potentially assigned moreresources, while other types of traffic may be throttled or blocked.Assigned resources and/or policies may be variable and may changethroughout the day, throughout the year, based on congestion, serviceagreements and the like.

The vast majority of connection-based or session-based traffic on theInternet today makes use of the Transmission Control Protocol (TCP). TCPis popular at least in part because it generally provides reliable andfair delivery of data. That is, the information that is sent by a senderis eventually received at a receiver and no one sender that adheres togenerally adopted fairness aspects of Internet protocols can utilizemore than their fair share of the bandwidth on average. However, eventhough TCP has evolved over the last decades, there are aspects of theprotocol that are not well matched to the characteristics, and moreparticularly to the variability of characteristics, of today's Internet.As examples, primary reliance on retransmission and use of windowingalgorithms for congestion control are not always well matched to thereal-time requirements and dynamic nature of communication channels thatmay have relatively rapidly varying characteristics, for example,periodic delay of the delivery of packets or rapidly changing linkcapacity.

As a result, applications running over today's Internet may be plaguedby long delays in transferring large data files, by pauses orinterruptions in video or audio streaming, by low audio or picturequality and/or by slow response times for real-time interactive content.These problems may be accompanied by and/or the result of an inefficientuse of the underlying network bandwidth due to overly restrictivecongestion control and/or to the large numbers of end-to-end packetretransmissions.

One technology that has been proposed to address some of the limitationsof TCP for communication over today's Internet is Random Linear NetworkCoding (RLNC), which involves a combination of using random linear codesfor error correction and recoding of packets at intermediate nodes inthe network. However, RLNC alone has not provided a complete solution tolimitations imposed by network characteristics. Other proposedtechnologies based on new codes, forward error correction codes, dataencryption techniques, and the like, also have not been shown to providecomplete solutions. Therefore, there is a need for a new protocol toensure high-speed uninterrupted delivery of data packets over networksthat comprises many different types of equipment, operated by manydifferent operators, over many different types of wired and wirelesslinks.

Also proposed has been the use of the user datagram protocol (UDP),which can speed up data delivery times but at the expense of reliabledata delivery. While some users and/or applications may be able totolerate lost and/or out-of-order data packets at a receiver, networkoperators have been known to impose policies that limit or block theamount of UDP traffic that may flow over their networks at any giventime. These restrictions are at least partially motivated by the factthat many of the current proprietary protocols running over UDP arebelieved to be unfair, meaning they may consume as much bandwidth and/ornetwork resources as they can in order to deliver their data veryquickly.

Thus there is a need for a new protocol that can reliably deliver datapackets over today's Internet faster than TCP but not at the expense offairness.

SUMMARY

In a general aspect, a method for data communication between a firstnode and a second node over a data path coupling the first node and thesecond node, includes determining one or more redundancy messages fromdata messages at the first node using an error correcting code,transmitting messages from the first node to the second node over thedata path, the transmitted messages including the data messages and theone or more redundancy messages, receiving messages at the first nodefrom the second node, including receiving messages indicative of a rateof arrival at the second node of the messages transmitted from the firstnode, and receiving messages indicative of successful and unsuccessfuldelivery of the messages transmitted from the first node to the secondnode, maintaining a first transmission limit according to the receivedmessages, maintaining a second transmission limit according to thereceived messages; and inhibiting transmission of messages from thefirst node to the second node, including limiting transmission ofmessages according to the maintained first transmission limit, andaccording to the second transmission limit.

Aspects may include one or more of the following features.

Maintaining the first transmission limit may include maintaining atransmission rate limit, and wherein limiting transmission of themessages according to the first transmission limit includes limiting atransmission rate of said messages. Maintaining the second transmissionlimit may include maintaining a window size, and wherein limitingtransmission of the messages according to the second transmission limitmay include limiting a number of messages not yet acknowledged assuccessfully delivered to the second node according to the window size.The window size may include a congestion control window size.

Receiving messages indicative of the rate of arrival may includereceiving acknowledgment messages from the second node, wherein a rateof arrival of said acknowledgment messages may be indicative of the rateof arrival of the messages at the second node. The rate of arrival ofthe acknowledgment may include a rate of acknowledgment of packets. Atleast some of the messages indicative of a rate of arrival and some ofthe messages indicative of successful and unsuccessful delivery of themessage may be same messages. The error correcting code may be a linearerror correcting code.

In another general aspect, a communication apparatus includes a firstdevice having an interface for passing messages to and from a seconddevice over a communication path coupling the first device to the seconddevice. The first device includes a communication controller configuredto determine one or more redundancy messages from data messages at thefirst node using an error correcting code, transmit messages from thefirst node to the second node over the data path, the transmittedmessages including the data messages and the one or more redundancymessages, receive messages at the first node from the second node,including receiving messages indicative of a rate of arrival at thesecond node of the messages transmitted from the first node, andreceiving messages indicative of successful and unsuccessful delivery ofthe messages transmitted from the first node to the second node,maintain a first transmission limit according to the received messagesmaintain a second transmission limit according to the received messages,and inhibit transmission of messages from the first node to the secondnode, including limiting transmission of messages according to themaintained first transmission limit, and according to the secondtransmission limit.

Aspects may include one or more of the following features.

Maintaining the first transmission limit may include maintaining atransmission rate limit, and wherein limiting transmission of themessages according to the first transmission limit may include limitinga transmission rate of said messages. Maintaining the secondtransmission limit may include maintaining a window size, and whereinlimiting transmission of the messages according to the secondtransmission limit may include limiting a number of messages not yetacknowledged as successfully delivered to the second node according tothe window size. The window size may include a congestion control windowsize. Receiving messages indicative of the rate of arrival may includereceiving acknowledgment messages from the second node, wherein a rateof arrival of said acknowledgment messages may be indicative of the rateof arrival of the messages at the second node. The rate of arrival ofthe acknowledgments may include a rate of acknowledgment of packets. Atleast some of the messages indicative of a rate of arrival and some ofthe messages indicative of successful and unsuccessful delivery of themessage may be same messages. The error correcting code may be a linearerror correcting code.

In a general aspect, a method for data communication between a firstnode and a second node over a data path coupling the first node and thesecond node includes determining one or more redundancy messages fromdata messages at the first node using an error correcting code,transmitting messages from the first node to the second node over thedata path, the transmitted messages including the data messages andredundancy messages, receiving messages at the first node from thesecond node, including receiving messages indicative of successful andunsuccessful delivery of the messages transmitted from the first node tothe second node, maintaining an estimate of a rate at which loss eventsoccur over the communication path based on the messages received fromthe second node, including updating the estimate to incorporate a singleloss event when one or more of the messages received from the secondnode indicate an unsuccessful delivery of a single packet to the seconddata node, and updating the estimate to incorporate a single loss eventwhen one or more of the messages received from the second node indicatean unsuccessful delivery of a number of consecutively transmittedpackets to the second data node, and adjusting a rate of redundancymessages transmitted from the first node based on the estimate of therate at which loss events occur.

Aspects may include one or more of the following features.

Adjusting the rate of redundancy messages transmitted from the firstnode based on the estimate of the rate at which loss events occur mayinclude adjusting a ratio of the rate of redundancy messages transmittedfrom the first node to the estimate of the rate at which loss eventsoccur. The error correcting code used to determine the one or moreredundancy messages may be chosen based at least in part on an estimatedrate of loss events where a number of consecutive messagesunsuccessfully delivered to the second data node is less than apredetermined threshold. The second error correcting code may include aburst error correcting code.

In another general aspect, a communication apparatus includes a firstdevice having an interface for passing messages to and from a seconddevice over a communication path coupling the first device to the seconddevice. The first device includes a communication controller configuredto determine one or more redundancy messages from data messages at thefirst node using an error correcting code, transmit messages from thefirst node to the second node over the data path, the transmittedmessages including the data messages and redundancy messages, receivemessages at the first node from the second node, including receivingmessages indicative of successful and unsuccessful delivery of themessages transmitted from the first node to the second node, maintain anestimate of a rate at which loss events occur over the communicationpath based on the messages received from the second node, includingupdating the estimate to incorporate a single loss event when one ormore of the messages received from the second node indicate anunsuccessful delivery of a single packet to the second data node, andupdating the estimate to incorporate a single loss event when one ormore of the messages received from the second node indicate anunsuccessful delivery of a number of consecutively transmitted packetsto the second data node, and adjusting a rate of redundancy messagestransmitted from the first node based on the estimate of the rate atwhich loss events occur.

Aspects may include one or more of the following features.

Adjusting the rate of redundancy messages transmitted from the firstnode based on the estimate of the rate at which loss events occur mayinclude adjusting a ratio of the rate of redundancy messages transmittedfrom the first node to the estimate of the rate at which loss eventsoccur. The error correcting code used to determine the one or moreredundancy messages may be chosen based at least in part on an estimatedrate of loss events where a number of consecutive messagesunsuccessfully delivered to the second data node is less than apredetermined threshold. The second error correcting code may include aburst error correcting code.

In a general aspect, a method for data communication from a first nodeto a second node over a data channel coupling the first node and thesecond node includes receiving messages at the first node from thesecond node, including receiving messages comprising data that depend atleast in part of characteristics of the channel coupling the first nodeand the second node, transmitting messages from the first node to thesecond node, including applying forward error correction according toparameters determined from the received messages, the parametersdetermined from the received messages including at least two of a blocksize, an interleaving factor, and a code rate.

Aspects may include one or more of the following features.

The method may include selecting parameters for forward error correctionfor transmission of messages from the first node to the second node. Theselecting of the parameters may be performed at the first node. Theselecting of the parameters may be performed at the second node. Thereceived messages from the second node may include data representing theparameters. The method may include selecting the parameters for forwarderror correction at the second node, and forming the data representingsaid parameters at the second node.

In another general aspect, a method for data communication from a firstnode to a second node over a data channel coupling the first node andthe second node includes transmitting messages from the second node tothe first node, including transmitting a message to the first nodecomprising data that depends at least in part on measured or expectedcharacteristics of the channel coupling the first node and the secondnode, receiving messages at the second node from the first node, themessages comprising forward error correction applied at the first nodeaccording to the data of the message transmitted from the second node tothe first node.

Aspects may include one or more of the following features.

The data that depends at least in part on measured or expectedcharacteristics may include data that specifies characteristics of theforward error correction. The data that specifies the characteristics ofthe forward error correction may include data specifying at least two ofa block size, an interleaving factor, a code rate, a pacing rate, and awindow size. The data that depends at least in part on measured orexpected characteristics may include data characterizing a pattern ofmessage erasure on the channel. The data characterizing a pattern ofmessage erasure may include data representing an erasure rate.

The method may include computing at the second node characteristics offorward error correction for application to messages at the first nodeprior to their transmission to the second node. Computing thecharacteristics of the forward error correction may include usingmeasured or expected characteristics of the channel coupling the firstnode and the second node. Computing the characteristics of the forwarderror correction may include using a state of a consumer of the messagereceived from the first node at the second node. The state may include astate related to an amount of buffered messages for the consumer of themessages.

In another general aspect, a communication apparatus includes a firstdevice having an interface for passing messages to and from a seconddevice over a communication path coupling the first device to the seconddevice. The communication apparatus is configured to receive messages atthe first device from the second device, including receiving messagescomprising data that depend at least in part of characteristics of thechannel coupling the first device and the second device and transmitmessages from the first device to the second device, including applyingforward error correction according to parameters determined from thereceived messages, the parameters determined from the received messagesincluding at least two of a block size, an interleaving factor, and acode rate.

Aspects may include one or more of the following features.

The apparatus may be configured to select parameters for forward errorcorrection for transmission of messages from the first device to thesecond device. The selecting of the parameters may be performed at thefirst device. The selecting of the parameters may be performed at thesecond device. The received messages from the second device may includedata representing the parameters. The communication apparatus may beconfigured to select the parameters for forward error correction at thesecond device, and form the data representing said parameters at thesecond device.

In another general aspect, a communication apparatus includes a firstdevice having an interface for passing messages to and from a seconddevice over a communication path coupling the first device to the seconddevice. The communication apparatus is configured to transmit messagesfrom the second device to the first device, including transmit a messageto the first device comprising data that depends at least in part onmeasured or expected characteristics of the channel coupling the firstdevice and the second device, and receive messages at the second devicefrom the first device, the messages comprising forward error correctionapplied at the first device according to the data of the messagetransmitted from the second device to the first device.

Aspects may include one or more of the following features.

The data that depends at least in part on measured or expectedcharacteristics may include data that specifies characteristics of theforward error correction. The data that specifies the characteristics ofthe forward error correction may include data specifying at least two ofa block size, an interleaving factor, a code rate, a pacing rate, and awindow size. The data that depends at least in part on measured orexpected characteristics may include data characterizing a pattern ofmessage erasure on the channel. The data characterizing a pattern ofmessage erasure may include data representing an erasure rate.

The apparatus may be configured to compute at the second devicecharacteristics of forward error correction for application to messagesat the first device prior to their transmission to the second device.Computing the characteristics of the forward error correction mayinclude using measured or expected characteristics of the channelcoupling the first device and the second device. Computing thecharacteristics of the forward error correction may include using astate of a consumer of the message received from the first device at thesecond device. The state may include a state related to an amount ofbuffered messages for the consumer of the messages.

In a general aspect, a method for data communication between a firstnode and a second node over a number of data paths coupling the firstnode and the second node includes transmitting messages between thefirst node and the second node over the number of data paths includingtransmitting at least some of the messages over a first data path of thenumber of data paths using a first communication protocol, andtransmitting at least some of the messages over a second data path ofthe number of data paths using a second communication protocol anddetermining that the first data path is altering a flow of messages overthe first data path due to the messages being transmitted using thefirst communication protocol, and in response to the determining,adjusting a number of messages sent over the number of data pathsincluding decreasing a number of the messages transmitted over the firstdata path and increasing a number of messages transmitted over thesecond data path.

Aspects may include one or more of the following features.

Determining that the first data path is altering the flow of messagesover the first data path may include determining that the first datapath is limiting a rate of messages transmitted using the firstcommunication protocol. Determining that the first data path is alteringthe flow of messages over the first data path may include determiningthat the first data path is dropping messages transmitted using thefirst communication protocol at a higher rate than a rate at which thesecond data path is dropping messages transmitted using the secondcommunication protocol. The first communication protocol may be the UserDatagram Protocol (UDP). The second communication protocol may be theTransmission Control Protocol (TCP).

The messages may be initially equally divided across the first data pathand the second data path using a load balancing technique. The messagesmay be initially divided across the first data path and the second datapath according to a division of the messages across the first data pathand the second data path in one or more prior data communicationconnections. The messages may be initially divided across the first datapath and the second data path based on a probability that the first datapath will alter a flow of messages over the first data path due to themessages being transmitted using the first communication protocol.

The messages may be divided across the first data path and the seconddata path based on message type. The message type may include one ormore of acknowledgment messages, forward error correction messages,retransmission messages, and original data messages. Decreasing a numberof the messages transmitted over the first data path and increasing anumber of messages transmitted over the second data path may includesending all of the messages over the second path and sending none of themessages over the first path.

At least some of the number of data paths may share a common physicaldata path. The first data path and the second data path may share acommon physical data path. The adjusting of the number of messages sentover the number of data paths may occur during an initial phase of thetransmission of the messages. The adjusting of the number of messagessent over the number of data paths may repeatedly occur over a durationof the transmission of the messages. The adjusting of the number ofmessages sent over the number of data paths may include increasing anumber of the messages transmitted over the first data path anddecreasing a number of messages transmitted over the second data path.

In another general aspect, a system for data communication over a numberof data paths includes a first node and a second node coupled by thenumber of data paths and configured to transmit messages therebetweenover the number of data paths including transmitting at least some ofthe messages over a first data path of the number of data paths using afirst communication protocol, and transmitting at least some of themessages over a second data path of the number of data paths using asecond communication protocol and determine that the first data path isaltering a flow of messages over the first data path due to the messagesbeing transmitted using the first communication protocol, and inresponse to the determining, adjusting a number of messages sent overthe number of data paths including decreasing a number of the messagestransmitted over the first data path and increasing a number of messagestransmitted over the second data path.

Aspects may include one or more of the following features.

Determining that the first data path is altering the flow of messagesover the first data path may include determining that the first datapath is limiting a rate of messages transmitted using the firstcommunication protocol. Determining that the first data path is alteringthe flow of messages over the first data path may include determiningthat the first data path is dropping messages transmitted using thefirst communication protocol at a higher rate than a rate at which thesecond data path is dropping messages transmitted using the secondcommunication protocol. The first communication protocol may be the UserDatagram Protocol (UDP). The second communication protocol may be theTransmission Control Protocol (TCP).

The messages may be initially equally divided across the first data pathand the second data path using a load balancing technique. The messagesmay be initially divided across the first data path and the second datapath according to a division of the messages across the first data pathand the second data path in one or more prior data communicationconnections. The messages may be initially divided across the first datapath and the second data path based on a probability that the first datapath will alter a flow of messages over the first data path due to themessages being transmitted using the first communication protocol. Themessages may be divided across the first data path and the second datapath based on message type. The message type may include one or more ofacknowledgment messages, forward error correction messages,retransmission messages, and original data messages. Decreasing a numberof the messages transmitted over the first data path and increasing anumber of messages transmitted over the second data path may includesending all of the messages over the second path and sending none of themessages over the first path.

At least some of the number of data paths may share a common physicaldata path. The first data path and the second data path may share acommon physical data path. The adjusting of the number of messages sentover the number of data paths may occur during an initial phase of thetransmission of the messages. The adjusting of the number of messagessent over the number of data paths may repeatedly occur over a durationof the transmission of the messages. The adjusting of the number ofmessages sent over the number of data paths may include increasing anumber of the messages transmitted over the first data path anddecreasing a number of messages transmitted over the second data path.

In another general aspect, software stored on non-transitorycomputer-readable media comprising instructions for causing one or moreprocessors to execute a data communication method for data communicationbetween a number of nodes over a number of data paths coupling thenumber of nodes including transmitting messages between the first nodeand the second node over the number of data paths including transmittingat least some of the messages over a first data path of the number ofdata paths using a first communication protocol, and transmitting atleast some of the messages over a second data path of the number of datapaths using a second communication protocol and determining that thefirst data path is altering a flow of messages over the first data pathdue to the messages being transmitted using the first communicationprotocol, and in response to the determining, adjusting a number ofmessages sent over the number of data paths including decreasing anumber of the messages transmitted over the first data path andincreasing a number of messages transmitted over the second data path.

Aspects may have one or more of the following advantages.

In some examples, the parallel transmission over TCP and UDP is handleddifferently from conventional load balancing techniques because TCP andUDP 1) share a low-level data path and 2) have very different protocolcharacteristics.

In some examples, approaches respond to instantaneous network behaviorand learn the network's data handling policy and state by probing forchanges. Unlike conventional load-balancers which assume each data pathis unique and does not affect the other, approaches recognize that TCPand UDP share a low-level data path and directly affect each other.Additionally, TCP provides in-order delivery and retransmits data (alongwith flow control, congestion control, etc.) whereas UDP does not. Thisuniqueness requires additional logic that maps specific message types toeach communication protocol based at least in part on the differentproperties of the protocols (e.g. expect longer jitter over TCP, expectout-of-order delivery on UDP). For example, the system does not codeover packets sent through TCP since it is reliable, but sends forwarderror correction exclusively over UDP to add redundancy and savebandwidth. In some examples, a larger ACK interval is used for ACKingTCP data.

By employing the techniques described herein, approaches distribute dataover TCP and UDP data paths to achieve optimal or near-optimalthroughput when network provider's policies treat UDP unfairly ascompared to conventional systems, which simply use UDP if possible andfallback to TCP if not.

In a general aspect, a method for data communication between a firstnode and a second node over a data path coupling the first node and thesecond node including transmitting messages from the first node to thesecond node over the data path, receiving messages at the first nodefrom the second node, including receiving messages indicative ofsuccessful and unsuccessful delivery of the messages transmitted fromthe first node to the second node, maintaining a transmission limitaccording to the received messages indicative of successful andunsuccessful delivery of messages, the maintaining including decreasingthe transmission limit when the received messages indicate anunsuccessful delivery of a message transmitted from the first node tothe second node, increasing the transmission limit according to anincrease function while the received messages indicate that no messageswere unsuccessfully delivered to the second node, and wherein theincrease function includes a first parameter for controlling a shape ofa first portion of the increase function and a second parameter forcontrolling a shape of a second portion of the increase function, andinhibiting transmission of messages from the first node to the secondnode, including limiting transmission of messages according to themaintained transmission limit.

Aspects may include one or more of the following features.

Maintaining the second transmission limit may include maintaining awindow size, and limiting transmission of the messages according to thesecond transmission limit may include limiting a number of messages notyet successfully delivered to the second node according to the windowsize. The window size may include a congestion control window size.Decreasing the transmission limit may include decreasing the windowsize. Increasing the transmission limit may include increasing thewindow size. The first portion of the increase function may have aconvex shape. The second portion of the increase function may have aconcave shape. The first portion of the increase function may be definedas:

W ₁ =W _(max) +c ₁(t−k)³

where W_(max) is a transmission limit threshold, c₁ is the firstparameter, and k is defined as:

${k = \sqrt[3]{\frac{W_{\max} - W}{c_{1}}}}.$

The second portion of the increase function may be defined as

W ₂(t)=W _(max) +c ₂(t−k)³

where W_(max) is a transmission limit threshold, c₂ is the secondparameter, and k is defined as:

${k = \sqrt[3]{\frac{W_{\max} - W}{c_{1}}}}.$

The first portion of the increase function may be used to increasetransmission limit up to a transmission limit threshold and the secondportion of the increase function may be used to increase thetransmission limit beyond the transmission limit threshold.

In another general aspect, a communication apparatus includes a firstdevice having an interface for passing messages to and from a seconddevice over a communication path coupling the first device to the seconddevice. The first device includes a communication controller configuredto transmit messages from the first node to the second node over thedata path, receive messages at the first node from the second node,including receive messages indicative of successful and unsuccessfuldelivery of the messages transmitted from the first node to the secondnode, maintain a transmission limit according to the received messagesindicative of successful and unsuccessful delivery of messages, themaintaining including decreasing the transmission limit when thereceived messages indicate an unsuccessful delivery of a messagetransmitted from the first node to the second node, increasing thetransmission limit according to an increase function while the receivedmessages indicate that no messages were unsuccessfully delivered to thesecond node, and wherein the increase function includes a firstparameter for controlling a shape of a first portion of the increasefunction and a second parameter for controlling a shape of a secondportion of the increase function, and inhibit transmission of messagesfrom the first node to the second node, including limiting transmissionof messages according to the maintained transmission limit.

Aspects may include one or more of the following features.

Maintaining the second transmission limit may include maintaining awindow size, and limiting transmission of the messages according to thesecond transmission limit may include limiting a number of messages notyet successfully delivered to the second node according to the windowsize. The window size may include a congestion control window size.Decreasing the transmission limit may include decreasing the windowsize. Increasing the transmission limit may include increasing thewindow size. The first portion of the increase function may have aconvex shape. The second portion of the increase function may have aconcave shape.

The first portion of the increase function may be defined as

W ₁(t)=W _(max) +c ₁(t−k)³

where W_(max) is a transmission limit threshold, c₁ is the firstparameter, and k is defined as:

$k = \sqrt[3]{\frac{W_{\max} - W}{c_{1}}}$

The second portion of the increase function may be defined as

W ₂(t)=W _(max) +c ₂(t−k)³

where W_(max) is a transmission limit threshold, c₂ is the secondparameter, and k is defined as:

$k = \sqrt[3]{\frac{W_{\max} - W}{c_{1}}}$

The first portion of the increase function may be used to increasetransmission limit up to a transmission limit threshold and the secondportion of the increase function may be used to increase thetransmission limit beyond the transmission limit threshold.

In a general aspect, a method for data communication between a firstnode and a second node over a data path coupling the first node and thesecond node includes transmitting a segment of data from the first nodeto the second node over the data path as a number of messages, thenumber of messages being transmitted according to a transmission order.A degree of redundancy associated with each message of the number ofmessages is determined based on a position of said message in thetransmission order.

Aspects may include one or more of the following features.

The degree of redundancy associated with each message of the number ofmessages may increase as the position of the message in the transmissionorder is non-decreasing. Determining the degree of redundancy associatedwith each message of the number of messages based on the position (i) ofthe message in the transmission order is further based on one or more ofdelay requirements for an application at the second node, a round triptime associated with the data path, a smoothed loss rate (P) associatedwith the channel, a size (N) of the data associated with the number ofmessages, a number (a_(i)) of acknowledgment messages received from thesecond node corresponding to messages from the number of messages, anumber (f_(i)) of in-flight messages of the number of messages, and anincreasing function (g(i)) based on the index of the data associatedwith the number of messages.

The degree of redundancy associated with each message of the number ofmessages may be defined as: (N+g(i)−a_(i))/(1-p)−f_(i). g(i) may bedefined as a maximum of a parameter m and N-i. g(i) may be defined asN−p(i) where p is a polynomial, with integer rounding as needed. Themethod may include receiving, at the first node, a feedback message fromthe second node indicating a missing message at the second node and, inresponse to receiving the feedback message, sending a redundancy messageto the second node to increase a degree of redundancy associated withthe missing message. The method may include maintaining, at the firstnode, a queue of preemptively computed redundancy messages and, inresponse to receiving the feedback message, removing some or all of thepreemptively computed redundancy messages from the queue and adding theredundancy message to the queue for transmission. The redundancy messagemay be generated and sent on-the-fly in response to receipt of thefeedback message.

The method may include maintaining, at the first node, a queue ofpreemptively computed redundancy messages for the number of messagesand, in response to receiving a feedback message indicating successfuldelivery of the number of messages, removing any preemptively computedredundancy messages associated with the number of messages from thequeue of preemptively computed redundancy messages. The degree ofredundancy associated with each of the messages may characterize aprobability of correctability of an erasure of the message. Theprobability of correctability may depend on a comparison of between thedegree of redundancy and a loss probability.

In another general aspect, a system for data communication between anumber of nodes over a data path coupling the number of nodes includes afirst node configured to transmit a segment of data to a second nodeover the data path as a number of messages, the number of messages beingtransmitted according to a transmission order. A degree of redundancyassociated with each message of the number of messages is determinedbased on a position of said message in the transmission order.

In another general aspect, software stored on non-transitorycomputer-readable media comprising instructions for causing one or moreprocessors to execute a data communication method for data communicationbetween a number of nodes over a data path coupling the number of nodesincluding transmitting a segment of data from the first node to thesecond node over the data path as a number of messages, the number ofmessages being transmitted according to a transmission order. A degreeof redundancy associated with each message of the number of messages isdetermined based on a position of said message in the transmissionorder.

In another general aspect, a method for data communication between afirst node and a second node over a data path coupling the first nodeand the second node includes transmitting messages between the firstnode and the second node over the data path, the messages including datamessages formed from a segment of data and redundancy messages formedfor the segment of data, the transmitting including maintaining, at thefirst node, a queue for storing the redundancy messages prior totransmission, receiving, at the first node, feedback messagescharacterizing a delivery status of the messages from the second node.Maintaining the queue of the redundancy messages includes addingredundancy messages to the queue or removing redundancy messages fromthe queue based on the received feedback messages.

Aspects may include one or more of the following features.

Receiving feedback messages may include receiving a messagecharacterizing a degree of successful delivery of the messages at thesecond node and, in response to receiving the feedback message, adding aredundancy message to the queue for transmission to the second node. Themethod may include removing some or all redundancy messages from thequeue prior to adding the redundancy message to the queue. Receivingfeedback messages may include receiving a feedback message from thesecond node indicating a degree of successful delivery of one or moremessages at the second node and, in response to receiving the feedbackmessage, removing any redundancy messages associated with the one ormore messages from the queue.

In another general aspect, a system for data communication between anumber of nodes over a data path coupling the number of nodes includes afirst node configured to transmit messages to a second node over thedata path, the messages including data messages formed from a segment ofdata and redundancy messages formed for the segment of data, thetransmitting including maintaining, at the first node, a queue forstoring the redundancy messages prior to transmission and receivefeedback messages characterizing a delivery status of the messages fromthe second node. Maintaining the queue of the redundancy messagesincludes adding redundancy messages to the queue or removing redundancymessages from the queue based on the received feedback messages.

In another general aspect, software stored on non-transitorycomputer-readable media comprising instructions for causing one or moreprocessors to execute a data communication method for data communicationbetween a number of nodes over a data path coupling the number of nodes,including transmitting messages between the first node and the secondnode over the data path, the messages including data messages formedfrom a segment of data and redundancy messages formed for the segment ofdata, the transmitting including maintaining, at the first node, a queuefor storing the redundancy messages prior to transmission, receiving, atthe first node, feedback messages characterizing a delivery status ofthe messages from the second node. Maintaining the queue of theredundancy messages includes adding redundancy messages to the queue orremoving redundancy messages from the queue based on the receivedfeedback messages.

In a general aspect, a method for data communication from a first nodeto a second node over a data channel coupling the first node and thesecond node includes receiving data messages at the second node, themessages belonging to a set of data messages transmitted in a sequentialorder from the first node, sending feedback messages from the secondnode to the first node, the feedback messages characterizing a deliverystatus of the set of data messages at the second node, includingmaintaining a set of one or more timers according to occurrences of anumber of delivery order events, the maintaining including modifying astatus of one or more timers of the set of timers based on occurrencesof the number of delivery order events, and deferring sending of saidfeedback messages until expiry of one or more of the set of one or moretimers.

Aspects may include one or more of the following features.

The set of one or more timers may include a first timer and the firsttimer may be started upon detection of a first delivery order event, thefirst delivery order event being associated with receipt of a first datamessage associated with a first position in the sequential order priorto receipt of one or more missing messages associated with positionspreceding the first position in the sequential order. The method mayinclude sending the feedback messages indicating a successful deliveryof the set of data messages at the second node upon detection of asecond delivery order event, the second delivery order event beingassociated with receipt of the one or more missing messages prior toexpiry of the first timer. The method may include sending said feedbackmessages indicating an unsuccessful delivery of the set of data messagesat the second node upon expiry of the first timer prior to any of theone or more missing messages being received. The set of one or moretimers may include a second timer and the second timer is started upondetection of a second delivery order event, the second delivery orderevent being associated with receipt of some but not all of the missingmessages prior to expiry of the first timer. The method may includesending feedback messages indicating an unsuccessful delivery of the setof data messages at the second node upon expiry of the second timerprior to receipt of the missing messages. The method may include sendingfeedback messages indicating a successful delivery of the set of datamessages at the second node upon detection of a third delivery orderevent, the third delivery order event being associated with receipt ofthe missing messages prior to expiry of the second timer.

In another general aspect, a system for data communication over a datachannel coupling a number of nodes includes a second node of the numberof nodes configured to receive data messages, the data messagesbelonging to a set of data messages transmitted in a sequential orderfrom a first node, send feedback messages to the first node, thefeedback messages characterizing a delivery status of the set of datamessages at the second node, including maintaining a set of one or moretimers according to occurrences of a number of delivery order events,the maintaining including modifying a status of one or more timers ofthe set of timers based on occurrences of the number of delivery orderevents, and deferring sending of said feedback messages until expiry ofone or more of the set of one or more timers.

In another general aspect, software stored on non-transitorycomputer-readable media including instructions for causing a second nodein a data communication system to receive data messages at the secondnode, the messages belonging to a set of data messages transmitted in asequential order from the first node, send feedback messages from thesecond node to the first node, the feedback messages characterizing adelivery status of the set of data messages at the second node,including maintaining a set of one or more timers according tooccurrences of a number of delivery order events, the maintainingincluding modifying a status of one or more timers of the set of timersbased on occurrences of the number of delivery order events, anddeferring sending of said feedback messages until expiry of one or moreof the set of one or more timers.

In another general aspect, a method for data communication from a firstnode to a second node over a data channel coupling the first node andthe second node includes receiving, at the first node, feedback messagesindicative of a delivery status of a set of data messages transmitted ina sequential order to the second node from the second node, maintaininga size of a congestion window at the first node including maintaining aset of one or more timers according to occurrences of a number offeedback events, the maintaining including modifying a status of one ormore timers of the set of timers based on occurrences of the number offeedback events, and delaying modification of the size of the congestionwindow until expiry of one or more of the set of one or more timers.

Aspects may include one or more of the following features.

The set of one or more timers may include a first timer and the firsttimer may be started upon detection of a first feedback event, the firstfeedback event being associated with receipt of a first feedback messageindicating successful delivery of a first data message having firstposition in the sequential order prior to receipt of one or morefeedback messages indicating successful delivery of one or more otherdata messages having positions preceding the first position in thesequential order. The method may include cancelling modification of thecongestion window upon detection of a second feedback event, the secondfeedback event being associated with receipt of one or more feedbackmessages indicating successful delivery of the one or more other datamessages prior to expiry of the first timer. The method may includemodifying the congestion window upon expiry of the first timer prior toreceipt of any feedback message indicating successful delivery of theone or more other data messages.

The set of one or more timers may include a second timer and the secondtimer may be started upon detection of a third feedback event, the thirdfeedback event being associated with receipt of one or more feedbackmessages indicating successful delivery of some but not all of the oneor more other data messages prior to expiry of the first timer. Themethod may include modifying the size of the congestion window uponexpiry of the second timer prior to receipt of one or more feedbackmessages indicating successful delivery of the one or more other datamessages. The method may include cancelling modification of the size ofthe congestion window upon detection of a fourth feedback event, thefourth feedback event being associated with receipt one or more feedbackmessages indicating successful delivery of the one or more other datamessages prior to expiry of the second timer.

In another general aspect, a system for data communication between anumber of nodes over a data channel coupling the number of nodesincludes a first node of the number of nodes configured to receive, atthe first node, feedback messages indicative of a delivery status of aset of data messages transmitted in a sequential order to the secondnode from the second node, maintain a size of a congestion window at thefirst node including maintaining a set of one or more timers accordingto occurrences of a number of feedback events, the maintaining includingmodifying a status of one or more timers of the set of timers based onoccurrences of the number of feedback events, and delaying modificationof the size of the congestion window until expiry of one or more of theset of one or more timers.

In another general aspect, software stored on non-transitorycomputer-readable media includes instructions for causing a first nodein a data communication system to receive, at the first node, feedbackmessages indicative of a delivery status of a set of data messagestransmitted in a sequential order to the second node from the secondnode, maintain a size of a congestion window at the first node includingmaintaining a set of one or more timers according to occurrences of anumber of feedback events, the maintaining including modifying a statusof one or more timers of the set of timers based on occurrences of thenumber of feedback events, and delaying modification of the size of thecongestion window until expiry of one or more of the set of one or moretimers.

In a general aspect, a method for data communication over a data channelon a data path between a first node and a second node includesmaintaining data characterizing one or more current or previous datacommunication connections traversing the data channel and initiating anew data communication connection between the first node and the secondnode including configuring the new data communication connection atleast in part according to the maintained data.

Aspects may include one or more of the following features.

The maintained data may characterize one or more data channels on one ormore data paths between the first node and the second node over whichsaid one or more current or previous data communication connectionspass. The maintained data may characterize an error rate of the one ormore data channels. The maintained data may characterize a bandwidth ofthe one or more data channels. The maintained data may characterize around trip time of the one or more data channels. The maintained datamay characterize communication protocol parameters of the one or morecurrent or previous data communication connections.

The communication protocol parameters may include one or more of acongestion window size, a block size, an interleaving factor, a portnumber, a pacing interval, a round trip time, and a timing variability.The communication protocol parameters may include two or more of acongestion window size, a block size, an interleaving factor, a portnumber, a pacing interval, a round trip time, and a timing variability.

The maintained data may characterize forward error correction parametersassociated with the one or more current or previous data communicationconnections. The forward error correction parameters may include a coderate. Initiating the new data communication connection may includeconfiguring the new data communication connection according to firstdata of the maintained data, the first data is maintained at the firstnode, and initiating the new data communication connection includesproviding the first data from the first node to the second node forconfiguring the new data communication connection.

Initiating the new data communication connection may include configuringthe new data communication connection according to first data of themaintained data, the first data is maintained at the first node, andinitiating the new data communication connection includes accessingfirst data at the first node for configuring the new data communicationconnection.

Initiating the new data communication connection may include configuringthe new data communication connection according to first data of themaintained data, the first data being maintained at the first node, andinitiating the new data communication connection includes accepting arequest from the first node for establishing the new data communicationconnection between the first node and the second node, includingreceiving, at the second node, at least one message from the first nodecomprising the first data for configuring said connection. The methodmay include maintaining the new data communication connection betweenthe first node and the second node, including maintaining communicationparameters, including initializing said communication parametersaccording the first data received in the at least one message from thefirst node.

Maintaining the new data communication connection may include adaptingthe communication parameters according to feedback from the first node.The feedback from the first node may include feedback messages receivedfrom the first node. The feedback may include feedback derived from aplurality of feedback messages received from the first node.

In another general aspect, a system for data communication over a datachannel on a data path between a first node and a second node includes adata store for maintaining data characterizing one or more current orprevious data communication connections traversing the data channel anda connection initiation module for initiating a new data communicationconnection between the first node and the second node includingconfiguring the new data communication connection at least in partaccording to the maintained data.

In another general aspect, software stored on non-transitorycomputer-readable media comprising instructions for causing one or moreprocessors to execute a data communication method for data communicationbetween a first node and a second node over a data path coupling thefirst node and the second node includes maintaining data characterizingone or more current or previous data communication connectionstraversing the data channel and initiating a new data communicationconnection between the first node and the second node includingconfiguring the new data communication connection at least in partaccording to the maintained data.

In some examples, one or more training communication connections over adata channel on a data path are employed prior to establishment of datacommunication connections over the data channel on the data path. Thetraining communication connections are used to collect information aboutthe data channel which is then used when establishing the datacommunication connections. In other examples, no training communicationconnections are employed and information about the data channel isobtained from one or more previous or current data communicationconnection over the data channel on the data path.

In a general aspect, a method for data communication between a firstnode and a second node over a number of data paths coupling the firstnode and the second node includes transmitting messages between thefirst node and the second node over the number of data paths includingtransmitting a first subset of the messages over a first data path ofthe number of data paths, and transmitting a second subset of themessages over a second data path of the number of data paths. The firstdata path has a first latency and the second data path has a secondlatency substantially larger than the first latency, and messages of thefirst subset of the messages are chosen to have first messagecharacteristics and messages of the second subset are chosen to havesecond message characteristics, different from the first messagecharacteristics.

Aspects may include one or more of the following features.

Messages having the first message characteristics may include timecritical messages. The first subset of the messages and the secondsubset of the messages may be determined from a portion of the messagesavailable at the first node at a time of transmission. At a subsequenttime of transmission, additional messages made available to the firstnode may be divided into the first subset and the second subset based onmessage characteristics associated with the additional messages.Messages having the first message characteristics may be associated withan initial subset of a data set and messages having the second messagecharacteristics may be associated with a subsequent subset of the dataset. The messages of the second subset may include messages that are atmost n messages ahead of a last acknowledged message in a sequentialtransmission order associated with the messages, wherein n is determinedbased on a buffer size at one of the first and second nodes.

Messages having the first message characteristics may includeacknowledgment messages and messages having the second messagecharacteristics may include data messages. Messages having the firstmessage characteristics may include supplemental data messages. Thesupplemental data messages include data messages may include redundancydata and messages having the second message characteristics includeoriginal data messages. The first data path may include a terrestrialdata path and the second data path may include a satellite data path.The terrestrial data path may include one or more of a cellular datapath, a digital subscriber line (DSL) data path, a fiber optic datapath, a cable internet based data path, and a wireless local areanetwork data path. The satellite data path may include one or more of alow earth orbit satellite data path, a medium earth orbit satellite datapath, and a geostationary earth orbit satellite data path. The firstdata path may include a medium earth orbit satellite data path or a lowearth orbit satellite data path and the second data path may include ageostationary orbit satellite data path.

The method may further include, for each path of the number of datapaths, maintaining an indication of successful and unsuccessful deliveryof the messages over the data path and adjusting a congestion window forthe data path based on the indication. The method may further include,for each path of the number of data paths, maintaining, at the firstnode, an indication of whether a number of messages received at thesecond node is sufficient to decode data associated with the messages,wherein the indication is based on feedback received at the first nodeover the number of data paths.

In another general aspect, a system for data communication between anumber of nodes over a number of data paths coupling the number of nodesincludes a first node configured to transmit messages between to asecond node over the number of data paths including transmitting a firstsubset of the messages over a first data path of the number of datapaths, and transmitting a second subset of the messages over a seconddata path of the number of data paths. The first data path has a firstlatency and the second data path has a second latency substantiallylarger than the first latency, and messages of the first subset of themessages are chosen to have first message characteristics and messagesof the second subset are chosen to have second message characteristics,different from the first message characteristics.

Aspects may include one or more of the following features.

Messages having the first message characteristics may include timecritical messages. The first subset of the messages and the secondsubset of the messages may be determined from a portion of the messagesavailable at the first node at a time of transmission. At a subsequenttime of transmission, additional messages made available to the firstnode may be divided into the first subset and the second subset based onmessage characteristics associated with the additional messages.Messages having the first message characteristics may be associated withan initial subset of a data set and messages having the second messagecharacteristics may be associated with a subsequent subset of the dataset.

The messages of the second subset may include messages that are at mostn messages ahead of a last acknowledged message in a sequentialtransmission order associated with the messages, wherein n is determinedbased on a receive buffer size at the second node. Messages having thefirst message characteristics may include acknowledgment messages andmessages having the second message characteristics may include datamessages. Messages having the first message characteristics may includesupplemental data messages. The supplemental data messages may includedata messages including redundancy data and messages having the secondmessage characteristics may include original data messages.

The first data path may include a terrestrial data path and the seconddata path may include a satellite data path. The terrestrial data pathmay include one or more of a cellular data path, a digital subscriberline (DSL) data path, a fiber optic data path, a cable internet baseddata path, and a wireless local area network data path. The satellitedata path may include one or more of a low earth orbit satellite datapath, a medium earth orbit satellite data path, and a geostationaryearth orbit satellite data path. The first data path may include amedium earth orbit satellite data path or a low earth orbit satellitedata path and the second data path may include a geostationary orbitsatellite data path.

The first node may be further configured to, for each path of the numberof data paths, maintain an indication of successful and unsuccessfuldelivery of the messages over the data path and adjust a congestionwindow for the data path based on the indication. The first node may befurther configured to maintain an aggregate indication of whether anumber of messages received at the second node over the number of datapaths is sufficient to decode data associated with the messages and totransmit supplemental messages based on the aggregate indication,wherein the aggregate indication is based on feedback from the secondnode received at the first node over the number of data paths.

In another general aspect, software stored on non-transitorycomputer-readable media comprising instructions for causing one or moreprocessors to execute a data communication method for data communicationbetween a number of nodes over a data path coupling the number of nodesincluding transmitting messages between a first node and a second nodeover the number of data paths including transmitting a first subset ofthe messages over a first data path of the number of data paths, andtransmitting a second subset of the messages over a second data path ofthe number of data paths. The first data path has a first latency andthe second data path has a second latency substantially larger than thefirst latency, and messages of the first subset of the messages arechosen to have first message characteristics and messages of the secondsubset are chosen to have second message characteristics, different fromthe first message characteristics.

In a general aspect, a method for modifying redundancy informationassociated with encoded data passing from a first node to a second nodeover a number of data paths includes receiving first encoded dataincluding first redundancy information at an intermediate node from thefirst node via a first channel connecting the first node and theintermediate node, the first channel having first channelcharacteristics and transmitting second encoded data including secondredundancy information from the intermediate node to the second node viaa second channel connecting the intermediate node and the second node,the second channel having second channel characteristics. A degree ofredundancy associated with the second redundancy information isdetermined by modifying the first redundancy information based on one orboth of the first channel characteristics and the second channelcharacteristics without decoding the first encoded data.

Aspects may include one or more of the following features.

Modifying the first redundancy information may include adding redundancyinformation to the first redundancy information. Modifying the firstredundancy information may include removing redundancy information fromthe first redundancy information. The second redundancy information maybe further formed by modifying the first redundancy information based onfeedback from the second node indicative of successful or unsuccessfuldelivery of the encoded data to the second node. The first encoded dataand the second encoded data may be encoded using a random linear networkcode. Modifying the first redundancy information based on one or both ofthe first channel characteristics and the second channel characteristicsmay include modifying the first redundancy information based on one ormore of a block size, a congestion window size, and a pacing rateassociated with the first channel characteristics and/or the secondchannel characteristics.

The method may include sending a feedback message from the intermediatenode to the first node acknowledging receipt of one or more messages atthe intermediate node. The method may include receiving a feedbackmessage from the second node at the intermediate node and, in responseto receiving the feedback message, transmitting additional redundancyinformation to the second node.

In another general aspect, a system for modifying redundancy informationassociated with encoded data passing from a first node to a second nodeover a number of data paths includes an intermediate node configured toreceive first encoded data including first redundancy information fromthe first node via a first channel connecting the first node and theintermediate node, the first channel having first channelcharacteristics and transmit second encoded data including secondredundancy information from the intermediate node to the second node viaa second channel connecting the intermediate node and the second node,the second channel having second channel characteristics. A degree ofredundancy associated with the second redundancy information isdetermined by modifying the first redundancy information based on one orboth of the first channel characteristics and the second channelcharacteristics without decoding the first encoded data.

Aspects may include one or more of the following features.

Modifying the first redundancy information at the intermediate node mayinclude adding redundancy information to the first redundancyinformation. Modifying the first redundancy information at theintermediate node may include removing redundancy information from thefirst redundancy information. The second redundancy information may befurther formed by the intermediate node by modifying the firstredundancy information based on feedback from the second node indicativeof successful or unsuccessful delivery of the encoded data to the secondnode. The first encoded data and the second encoded data may be encodedusing a random linear network code. Modifying the first redundancyinformation at the intermediate node based on one or both of the firstchannel characteristics and the second channel characteristics mayinclude modifying the first redundancy information based on one or moreof a block size, a congestion window size, and a pacing rate associatedwith the first channel characteristics and/or the second channelcharacteristics.

The system may send a feedback message from the intermediate node to thefirst node acknowledging receipt of one or more messages at theintermediate node. The system may receive a feedback message from thesecond node at the intermediate node and, in response to receiving thefeedback message, transmit additional redundancy information to thesecond node.

In another general aspect, software stored on non-transitorycomputer-readable media includes instructions for causing anintermediate node in a data communication system to modify redundancyinformation associated with encoded data passing from a first node to asecond node over a number of data paths including receiving firstencoded data including first redundancy information at the intermediatenode from the first node via a first channel connecting the first nodeand the intermediate node, the first channel having first channelcharacteristics and transmitting second encoded data including secondredundancy information from the intermediate node to the second node viaa second channel connecting the intermediate node and the second node,the second channel having second channel characteristics. A degree ofredundancy associated with the second redundancy information isdetermined by modifying the first redundancy information based on one orboth of the first channel characteristics and the second channelcharacteristics without decoding the first encoded data.

DESCRIPTION OF DRAWINGS

FIG. 1 is a schematic of a data network including server and clientnodes coupled by intermediate networks;

FIG. 2 is a block diagram illustrating the modules that implementTCP-based communication between a client node and a server node;

FIG. 3 is a block diagram illustrating the modules that implement PacketCoding Transmission Communication Protocol (PC-TCP) based communicationbetween a client node and a server node;

FIG. 4 is a schematic diagram of a use of the approach shown in FIG. 3for communication between a server and a module device on a cellularnetwork;

FIG. 5 is a block diagram of a PC-TCP module that uses a conventionalUDP module;

FIG. 6 is a block diagram of a PC-TCP module that is partiallyintegrated into a client application and partially implemented using aconventional UDP module;

FIG. 7 is a block diagram of a PC-TCP module that is split with userspace and kernel space components;

FIG. 8 is a block diagram of a proxy architecture;

FIG. 9 is a block diagram of a PC-TCP based proxy architecture in whicha proxy node communicates using both PC-TCP and conventional TCP;

FIG. 10 is a block diagram of a proxy-based architecture of FIG. 9embodied using a gateway device;

FIG. 11 is a block diagram of an alternative proxy architecture embodiedwithin a client node;

FIG. 12 is a block diagram of a second PC-TCP based proxy architecturein which a proxy node communicates using both PC-TCP and conventionalTCP;

FIG. 13 is a block diagram of a proxy-based architecture of FIG. 12embodied using a wireless access device;

FIG. 14 is a block diagram of a proxy-based architecture of FIG. 12embodied cellular network;

FIG. 15 is a block diagram of a proxy-based architecture of FIG. 12embodied cable television based data network;

FIG. 16 is a block diagram of an intermediate proxy that communicateswith a client node and with a server node using separate PC-TCPconnections;

FIG. 17 is a block diagram of a proxy-based architecture of FIG. 16embodied in a network device;

FIG. 18 is a block diagram of an intermediate proxy that recodescommunication between a client node and with a server node;

FIGS. 19-20 are diagrams that illustrate delivery of common content tomultiple destinations;

FIGS. 21A-K are schematic diagrams of various embodiments of PC-TCPcommunication approaches;

FIG. 22 is a block diagram of PC-TCP communication approach thatincludes window and rate control modules;

FIG. 23 is a schematic of a data network.

FIGS. 24-27 are block diagrams illustrating an embodiment PC-TCPcommunication approach that is configured according to a number oftunable parameters;

FIG. 28 is a diagram showing a network communication system using theapproach of FIGS. 24-27;

FIG. 29 is a schematic diagram illustrating use of stored communicationparameters;

FIG. 30 is a schematic diagram illustrating a first embodiment ofmulti-path content delivery; and

FIG. 31A, FIG. 31B, and FIG. 31C are schematic diagrams illustrating asecond embodiment of multi-path content delivery.

Table of Contents CROSS-REFERENCE TO RELATED APPLICATIONS BACKGROUNDSUMMARY DESCRIPTION OF DRAWINGS Table of Contents4 DETAILED DESCRIPTION1 Overview 2 Architectures and applications  2.1 Transport layerarchitectures  2.1.1 Kernel implementation  2.1.2 Alternative softwareimplementations  2.2 Proxy architectures  2.2.1 Conventional proxy 2.2.2 First alternative proxy node  2.2.3 Integrated proxy  2.2.4Second alternative proxy node  2.3 Intermediate proxy  2.4 Recoding node 2.5 Multipath transmission  2.5.1 Single endpoint pair  2.5.2Distributed source  2.5.3 Distributed content delivery  2.6 Multicast 2.7 Further illustrative examples 3 Packet Coding (PC)  3.1 Datacharacteristics  3.2 Channel Characteristics  3.3 Inter-packet coding 3.3.1 Forward error correction and repair retransmission  3.3.2 Randomlinear coding  3.4 Batch transmission  3.5 Protocol parameters  3.6Transmission control  3.6.1 Congestion control  3.6.2 Transmission  3.7Error control  3.7.1 Packet reordering  3.7.2 Acknowledgments  3.8Parameter control  3.8.1 Initialization  3.8.2 Tunable coding  3.8.3Cross-session parameter control  3.9 Multi-path 4 Alternatives andimplementations What is claimed is: ABSTRACT

DETAILED DESCRIPTION 1 Overview

Various embodiments described in this document relate to communicationprotocols that improve aspects of communication between nodes on a datanetwork. These aspects include, for instance, average, worst case, orvariability in communication delay, channel utilization, and/or errorrate. These embodiments are primarily described in the context of packetswitched networks, and more particularly in the context of InternetProtocol (IP) based packet switched networks. However, it should beunderstood that at least some of the embodiments are more generallyapplicable to data communication that does not use packet switching orIP, for instance based on circuit-switched of other forms of datanetworks.

Furthermore, various embodiments are described in the context of databeing sent from a “server” to a “client.” It should be understood thatthese terms are used very broadly, roughly analogous to “data source”and “data destination”. Furthermore, in at least some applications ofthe techniques, the nodes are peers, and may alternate roles as “server”and “client” or may have both roles (i.e., as data source and datadestination) concurrently. However, for the sake of exposition, exampleswhere there is a predominant direction of data flow from a “server” nodeto a “client” node are described with the understanding that thetechniques described in these examples are applicable to many othersituations.

One example for a client-server application involves a server passingmultimedia (e.g., video and audio) data, either recorded or live, to aclient for presentation to a user. Improved aspects of communicationfrom the client to the server in such an example can reducedcommunication delay, for instance providing faster startup, reducedinstances of interrupted playback, reduced instances of bandwidthreduction, and/or increased quality by more efficient channelutilization (e.g., by avoiding use of link capacity in retransmissionsor unnecessary forward error correction). This example is useful forexposition of a number of embodiments. However, it must be recognizedthat this is merely one of many possible uses of the approachesdescribed below.

FIG. 1 shows a high-level block diagram of some components that may beinterconnected on a portion of a data network. A general example of acommunication connection or session arranged on today's Internet may berepresented as a client node 120 (e.g., a client computer) communicatingwith a server node 110 (e.g., a server computer) over one network or aninterconnection of multiple networks 151-152. For example, the clientand server nodes may communicate over the public Internet using theInternet Protocol (IP).

Referring to FIG. 2, in an example involving conventional communicationtechniques, a client node 120 hosts a client application 222, whichcommunicates with a TCP module 226 that implements a TransmissionControl Protocol (TCP). The TCP module 226 communicates with an IPmodule 228 that implements an Internet Protocol for communicatingbetween nodes on the interconnection of networks. The communicationpasses between nodes of the networks over a channel 230 (i.e., anabstraction of the path comprising physical links between equipmentinterconnecting the nodes of the network). Similarly, the server node110 hosts a server application 212, a TCP module 216, and an IP module218. When the server application 110 and the client application 222communicate, for example, with data being passed from the serverapplication to the client application, TCP module 216 at the server node110 and the TCP layer 226 at the client node 120 interact to implementthe two endpoints for the Transmission Control Protocol (TCP).

Generally, data units 201 (e.g., encoding of multimedia frames or otherunits of application data) generated by the server application 212 arepassed to the TCP module 216. The TCP module assembles data payloads202, for example, concatenating multiple data units 201 and/or bydividing data units 201 into multiple data payloads 202. In thediscussion below, these payloads are referred to in some instances asthe “original” or “uncoded” “packets” or original or uncoded “payloads”,which are communicated to the client (i.e., destination) node in thenetwork. Therefore, it should be understood that the word “packet” isnot used with any connotation other than being a unit of communication.In the TCP embodiment illustrated in FIG. 2, each data payload 202 is“wrapped” in a TCP packet 204, which is passed to the IP module 218,which further wraps the TCP packet 204 in an IP packet 206 fortransmission from the server node 110 to the client node 120, over whatis considered to be a IP layer channel 230 linking the server node 110and the client node 120. Note that at lower layers, such as at a datalink layer, further wrapping, unwrapping, and/or rewrapping of the IPpacket 206 may occur, however, such aspects are not illustrated in FIG.2. Generally, each payload 202 is sent in at least one TCP packet 204and a corresponding IP packet 206, and if not successfully received bythe TCP module 226 at the client node 120, may be retransmitted again bythe TCP module 216 at the server node 110 to result in successfuldelivery. The data payloads 202 are broken down into the data units 201originally provided by the server application 212 and are then deliveredin the same order to the client application 222 as they were provided bythe server application 212.

TCP implements a variety of features, including retransmission of lostpackets, maintaining order of packets, and congestion control to avoidcongestion at nodes or links along the path through the network and toprovide fair allocation of the limited bandwidth between and within thenetworks at intermediate nodes. For example, TCP implements a “windowprotocol” in which only a limited number (or range of sequence numbers)of packets are permitted to be transmitted for which end-to-endacknowledgments have not yet been received. Some implementations of TCPadjust the size of the window, for example, starting initially with asmall window (“slow start”) to avoid causing congestion. Someimplementations of TCP also control a rate of transmission of packets,for example, according to the round-trip-time and the size of thewindow.

The description below details one or more alternatives to conventionalTCP-based communication as illustrated in FIG. 2. In general, thesealternatives improve one or more performance characteristics, forexample, one or more of overall throughput, delay, and jitter. In someapplications, these performance characteristics are directly related toapplication level performance characteristics, such as image quality ina multimedia presentation application. Referring to FIG. 1, in a numberof examples, these alternatives are directed to improving communicationbetween a server node 110 and at least one client node 120. One exampleof such communication is streaming media from the server node 110 to theclient nodes 120, however, it should be recognized that this is only oneof many examples where the described alternatives can be used.

It should also be understood that the network configuration illustratedin FIG. 1 is merely representative of a variety of configurations. Anumber of these configurations may have paths with disparatecharacteristics. For example, a path from the server node 110 to aclient node 120 may pass over links using different types of equipmentand with very different capacities, delays, error rates, degrees ofcongestion, etc. In many instances, it is this disparity that presentschallenges to achieving end-to-end communication that achieves highrate, low delay and/or low jitter. As one example, the client node 120may be a personal communication device on a wireless cellular network,the network 152 in FIG. 1 may be a cellular carrier's private wirednetwork, and network 151 may be the public Internet. In another example,the client node 120 may be a “WiFi” node of a private wireless localarea network (WLAN), network 152 may be a private local area network(LAN), and network 151 may be the public Internet.

A number of the alternatives to conventional TCP make use of a PacketCoding (PC) approach. Furthermore, a number of these approaches make useof Packet Coding essentially at the Transport Layer. Although differentembodiments may have different features, these implementations aregenerically referred to below as Packet Coding Transmission ControlProtocol (PC-TCP). Other embodiments are also described in which thesame or similar PC approaches are used at other layers, for instance, ata data link layer (e.g., referred to as PC-DL), and therefore it shouldbe understood that in general features described in the context ofembodiments of PC-TCP may also be incorporated in PC-DL embodiments.

Before discussing particular features of PC-TCP in detail, a number ofembodiments of overall system architectures are described. The laterdescription of various embodiments of PC-TCP should be understood to beapplicable to any of these system architectures, and others.

2 Architectures and Applications

2.1 Transport layer architectures2.1.1 Kernel implementation

Referring to FIG. 3, in one architecture, the TCP modules at the servernode 110 and the client node 120 are replaced with PC-TCP modules 316and 326, respectively. Very generally, the PC-TCP module 316 at theserver accepts data units 201 from the server application 212 and formsoriginal data payloads 202 (i.e., “uncoded packets”, formed internallyto the PC-TCP module 316 and not illustrated). Very generally, thesedata payloads 202 are transported to and/or reconstructed at the PC-TCPmodule 326 at the client node 120, where the data units 201 areextracted and delivered to the client application 222 in the same orderas provided by the server application 212. As described in substantiallymore detail below, at least some embodiments of the PC-TCP modules makeuse of Random Linear Coding (RLC) for forming packets 304 fortransmission from the source PC-TCP module to the destination PC-TCPmodule, with each packet 304 carrying a payload 302, which for at leastsome packets 304 is formed from a combination of multiple originalpayloads 202. In particular, at least some of the payloads 202 areformed as linear combinations (e.g., with randomly generatedcoefficients in a finite field) of original payloads 202 to implementForward Error Correction (FEC), or as part of a retransmission or repairapproach in which sufficient information is not provided using FEC toovercome loss of packets 304 on the channel 230. Furthermore, the PC-TCPmodules 316 and 326 together implement congestion control and/or ratecontrol to generally coexist in a “fair” manner with other transportprotocols, notably conventional TCP.

One software implementation of the PC-TCP modules 316 or 326, issoftware modules that are integrated into the operating system (e.g.,into the “kernel”, for instance, of a Unix-based operating system) inmuch the same manner that a conventional TCP module is integrated intothe operating system. Alternative software implementations are discussedbelow.

Referring to FIG. 4, in an example in which a client node 120 is asmartphone on a cellular network (e.g., on an LTE network) and a servernode 110 is accessible using IP from the client node, the approachillustrated in FIG. 3 is used with one end-to-end PC-TCP session linkingthe client node 120 and the server node 110. The IP packets 300 carryingpackets 304 of the PC-TCP session traverse the channel between the nodesusing conventional approaches without requiring any non-conventionalhandling between the nodes at the endpoints of the session.

2.1.2 Alternative Software Implementations

The description above includes modules generically labeled “PC-TCP”. Inthe description below, a number of different implementations of thesemodules are presented. It should be understood that, in general, anyinstance of a PC-TCP module may be implemented using any of thedescribed or other approaches.

Referring to FIG. 5, in some embodiments, the PC-TCP module 326 (or anyother instance of PC-TCP module discussed in this document) isimplemented as a PC-TCP module 526, which includes a Packet Coding (PC)module 525 that is coupled to (i.e., communicates with) a conventionUser Datagram Protocol (UDP) module 524. Essentially each PC-TCP packetdescribed above consists of a PC packet “wrapped” in a UDP packet. TheUDP module 524 then communicates via the IP modules in a conventionalmanner. In some implementations, the PC module 525 is implemented as a“user space” process, which communicates with a kernel space UDP module,while in other implementations, the PC module 525 is implement in kernelspace.

Referring to FIG. 6, in some embodiments, the PC module 625, or itsfunction, is integrated into a client application 622, which thencommunicates directly with the conventional UDP module 524. The PC-TCPmodule 626 therefore effectively spans the client application 622 andthe kernel implementation of the UDP module 524. While use of UDP tolink the PC modules at the client and at the server has certainadvantages, other protocols may be used. One advantage of UDP is thatreliable transmission through use of retransmission is not part of theUDP protocol, and therefore error handling can be carried out by the PCmodules.

Referring to FIG. 7, in some implementations, a PC-TCP module 726 isdivided into one part, referred to as a PC-TCP “stub” 727, whichexecutes in the kernel space, and another part, referred to as thePC-TCP “code” 728, which executes in the user space of the operatingsystem environment. The stub 727 and the code 728 communicate to providethe functionality of the PC-TCP module.

It should be understood that these software implementations are notexhaustive. Furthermore, as discussed further below, in someimplementations, a PC-TCP module of any of the architectures or examplesdescribed in this document may be split among multiple hosts and/ornetwork nodes, for example, using a proxy architecture.

2.2 Proxy Architectures 2.2.1 Conventional Proxy Node

Referring to FIG. 8, certain conventional communication architecturesmake use of proxy servers on the communication path between a clientnode 120 and a server node 110. For example, a proxy node 820 hosts aproxy server application 822. The client application 222 communicateswith the proxy server application 822, which acts as an intermediary incommunication with the server application 212 (not shown in FIG. 8). Itshould be understood that a variety of approaches to implementing such aproxy are known. In some implementations, the proxy application isinserted on the path without the client node necessarily being aware. Insome implementations, a proxy client 812 is used at the client node, insome cases forming a software “shim” between the application layer andthe transport layer of the software executing at the client node, withthe proxy client 812 passing communication to the proxy serverapplication. In a number of proxy approaches, the client application 222is aware that the proxy is used, and the proxy explicitly acts as anintermediary in the communication with the server application. Aparticular example of such an approach makes use of the SOCKS protocol,in which the SOCKS proxy client application (i.e., an example of theproxy client 812) communicates with a SOCKS proxy server application(i.e., an example of the proxy server application 822). The client andserver may communicate over TCP/IP (e.g., via TCP and IP modules 826 band 828 b, which may be implemented together in one TCP module), and theSOCKS proxy server application fulfills communication requests (i.e.,with the server application) on behalf of the client application (e.g.,via TCP and IP modules 826 a and 828 a). Note that the proxy serverapplication may also perform functions other than forwardingcommunication, for example, providing a cache of data that can be usedto fulfill requests from the client application.

2.2.2 First Alternative Proxy Node

Referring to FIG. 9, in an alternative proxy architecture, a proxy node920 hosts a proxy server application 922, which is similar to the proxyserver application 822 of FIG. 8. The client application 222communicates with the proxy server application 922, for example asillustrated using conventional TCP/IP, and in some embodiments using aproxy client 812 (e.g., as SOCKS proxy client), executing at the clientnode 120. As illustrated in FIG. 9, the proxy server application 922communicates with a server application using a PC-TCP module 926, whichis essentially the same as the PC-TCP module 326 shown in FIG. 3 forcommunicating with the PC-TCP module 316 at the server node 110.

In some embodiments, the communication architecture of FIG. 9 and theconventional communication architecture of FIG. 2 may coexist in thecommunication between the client application and the server applicationmay use PC-TCP, conventional TCP, or concurrently use both PC-TCP andTCP. The communication approach may be based on a configuration of theclient application and/or based on dialog between the client and serverapplications in establishing communication between them.

Referring to FIG. 10, in an example of the architecture shown in FIG. 9,the proxy application 922 is hosted in a gateway 1020 that links a localarea network (LAN) 1050 to the Internet. A number of conventional clientnodes 120 a-z are on the LAN, and make use of the proxy serverapplication to communicate with one or more server applications over theInternet. Various forms of gateway 1020 may be used, for instance, arouter, firewall, modem (e.g., cable modem, DSL modem, etc.). In suchexamples, the gateway 1020 may be configured to pass conventional TCP/IPcommunication between the client nodes 120 a-z and the Internet, and forcertain server applications or under certain conditions (e.g.,determined by the client, the server, or the gateway) use the proxy tomake use of PC-TCP for communication over the Internet.

It should be understood that the proxy architecture shown in FIG. 9 maybe equally applied to server nodes 110 that communicate with a proxynode using TCP/IP, with the proxy providing PC-TCP communication withclient nodes, either directly or via client side proxies. In such cases,the proxy server application serving the server nodes may be hosted, forinstance, in a gateway device, such as a load balancer (e.g., as mightbe used with a server “farm”) that links the servers to the Internet. Itshould also be understood that in some applications, there is a proxynode associated with the server node as well as another proxy associatedwith the client node.

2.2.3 Integrated Proxy

Referring to FIG. 11, in some examples, a proxy server application 1122,which provides essentially the same functionality as the proxy serverapplication 922 of FIG. 9, is resident on the client node 1120 ratherthan being hosted on a separate network node as illustrated in FIG. 9.In such an example, the connection between the client application 222and the proxy server application 1122 is local, with the communicationbetween them not passing over a data network (although internally it maybe passed via the IP software “stack”). For example, a proxy client 812(e.g., a SOCKS client) interacts locally with the proxy serverapplication 1122, or the functions of the proxy client 812 and the proxyserver application 1122 are integrated into a single software component.

2.2.4 Second Alternative Proxy Node

In examples of the first alternative proxy node approach introducedabove, communication between the client node and the proxy node usesconventional techniques (e.g., TCP/IP), while communication between theproxy node and the server node (or its proxy) uses PC-TCP. Such anapproach may mitigate congestion and/or packet error or loss on the linkbetween the server node and the proxy node, however, it would notgenerally mitigate issues that arise on the link between the proxy nodeand the client node. For example, the client node and the proxy node maybe linked by a wireless channel (e.g., WiFi, cellular, etc.), which mayintroduce a greater degree of errors than the link between the serverand the proxy node over a wired network.

Referring to FIG. 12, in a second proxy approach, the client node 120hosts a PC-TCP module 326, or hosts or uses any of the alternatives ofsuch a module described in this document. The client application 222makes use of the PC-TCP module 326 at the client node to communicationwith a proxy node 1220. The proxy node essentially translates betweenthe PC-TCP communication with the client node 120 and conventional(e.g., TCP) communication with the server node. The proxy node 1220includes a proxy server application 1222, which makes use of a PC-TCPmodule 1226 to communicate with the client node (i.e., forms transportlayer link with the PC-TCP module 326) at the client node, and uses aconventional TCP module 826 a to communicate with the server.

Examples of such a proxy approach are illustrated in FIGS. 13-15.Referring to FIG. 13, an example of a proxy node 1220 is integrated in awireless access device 1320 (e.g., a WiFi access point, router, etc.).The wireless access device 1320 is coupled to the server via a wiredinterface 1351 and coupled to a wireless client node 120 via a wirelessinterface 1352 at the access device and a wireless interface 1353 at theclient node. The wireless access device 1320 includes a proxy andcommunication stack implementation 1321, which includes the modulesillustrated for the proxy 1220 in FIG. 12, and the wireless client node120 includes an application and communication stack implementation 1322,which includes the modules illustrated for the client node 120 in FIG.12. Note that the IP packets 300 passing between the access device 1320and the client node 120 are generally further “wrapped” using a datalayer protocol, for example, in data layer packets 1350. As introducedabove, in some implementations, rather than implementing the PacketCoding at the transport layer, in a modification of the approach shownin FIG. 13, the Packet Coding approaches are implemented at the datalink layer.

Referring to FIG. 14, a proxy node 1220 is integrated in a node of aprivate land network of a cellular service provider. In this example,communication between a server 110 and the proxy node 1220 useconventional techniques (e.g., TCP) over the public Internet, whilecommunication between the proxy node and the client node use PC-TCP. Itshould be understood that the proxy node 1220 can be hosted at variouspoints in the service provider's network, including without limitationat a gateway or edge device that connects the provider's private networkto the Internet (e.g. a Packet Data Network Gateway of an LTE network),and/or at an internal node of the network (e.g., a serving gateway, basestation controller, etc.). Referring to FIG. 15, a similar approach maybe used with a cable television-based network. PC-TCP communication maypass between a head end device and a distribution network (e.g., afiber, coaxial, or hybrid fiber-coaxial network) to individual homes.For example, each home may have devices that include PC-TCP capabilitiesthemselves, or in some example, a proxy node (e.g., a proxy nodeintegrated in a gateway 1010 as shown in FIG. 10) terminates the PC-TCPconnections at each home. The proxy node that communicates with theserver 110 using conventional approaches, while communicating usingPC-TCP over the distribution network is hosted in a node in the serviceprovider's private network, for instance at a “head end” device 1220 bof the distribution network, or in a gateway device 1220 a that linksthe service provider's network with the public Internet.

2.3 Intermediate Proxy

Referring to FIG. 16, in another architecture, the channel between aserver node and a client node is broken into independent tandem PC-TCPlinks. An intermediate node 1620 has two instances of a PC-TCP module1626 and 1627. One PC-TCP module 1626 terminates a PC-TCP channel andcommunicates with a corresponding PC-TCP module at the server (e.g.,hosted at the server node or at a proxy associated with the servernode). The other PC-TCP module 1627 terminates a PC-TCP channel andcommunicates with a corresponding PC-TCP module at the client (e.g.,hosted at the client node or at a proxy associated with the clientnode). The two PC-TCP modules 1626 and 1627 are coupled via a routingapplication 1622, which passes decoded data units provided by one of thePC-TCP modules (e.g., module 1626 from the server node) and to anotherPC-TCP module for transmission to the client.

Note that parameters of the two PC-TCP channels that are bridged at theintermediate node 1620 do not have to be the same. For example, thebridged channels may differ in their forward error correction code rate,block size, congestion window size, pacing rate, etc. In cases in whicha retransmission protocol is used to address packet errors or lossesthat are not correctable with forward error correction coding, thePC-TCP modules at the intermediate node request or service suchretransmission requests.

In FIG. 16, only two PC-TCP modules are shown, but it should beunderstood that the intermediate node 1620 may concurrently provide alink between different pairs of server and client nodes.

Referring to FIG. 17, an example of this architecture may involve aserver node 110 communicating with an intermediate node 1620, forexample, hosted in a gateway device 1720 of a service provider networkwith the intermediate node 1620 also communicating with the client node120 via a second PC-TCP link.

2.4 Recoding Node

Referring to FIG. 18, another architecture is similar to the one shownin FIG. 16 in that an intermediate node 1820 is on a path between aserver node 110 and a client node 120, with PC-TCP communication passingbetween it and the server node and between it and the client node.

In FIG. 16, the PC-TCP modules 1626, 1627 fully decode and encode thedata passing through the node. In the approach illustrated in FIG. 18,such complete decoding is not necessary. Rather, a recoding PC-TCPmodule 1822 receives payloads 1802 a-b from PC-TCP packets 1804 a-b, andwithout decoding to reproduce the original uncoded payloads 202 (notshown), the module uses the received PC-TCP packets to send PC-TCPpackets 304, with coded payloads 302, toward the destination. Details ofvarious recoding approaches are described further later in thisdocument. However, in general, the processing by the recoding PC-TCPmodule includes one or more of the following functions: forwardingPC-TCP packets without modification to the destination; “dropping”received PC-TCP packets without forwarding, for example, if theredundancy provided by the received packets are not needed on theoutbound link; generating and transmitting new PC-TCP packets to provideredundancy on the outbound link. Note that the recording PC-TCP modulemay also provide acknowledgment information on the inbound PC-TCP link(e.g., without requiring acknowledgment from the destination node), forexample, to the server, and process received acknowledgments on theoutbound link. The processing of the received acknowledgments mayinclude causing transmission of additional redundant information in thecase that the originally provided redundancy information was notsufficient for reconstruction of the payload data.

In general, the recoding PC-TCP module maintains separate communicationcharacteristics on the inbound and outbound PC-TCP channels. Therefore,although it does not decode the payload data, it does provide controland, in general, the PC-TCP channels may differ in their forward errorcorrection code rate, block size, congestion window size, pacing rate,etc.

2.5 Multipath Transmission 2.5.1 Single Endpoint Pair

In examples described above, a single path links the server node 110 andthe client node 120. The possibility of using conventional TCPconcurrently with PC-TCP between two nodes was introduced. Moregenerally, communication between a pair of PC-TCP modules (i.e., one atthe server node 110 and one at the client node 120) may follow differentpaths.

Internet protocol itself supports packets passing from one node toanother following different paths and possibly being delivered out oforder. Multiple data paths or channels can link a pair of PC-TCP modulesand be used for a single session. Beyond native multi-path capabilitiesof IP networks, PC-TCP modules may use multiple explicit paths for aparticular session. For example, without intending to be exhaustive,combinations of the following types of paths may be used:

-   -   Uncoded TCP and PC over UDP    -   PC over conventional TCP and UDP    -   PC-TCP over wireless LAN (e.g., WIFI, 802.11) and cellular data        (e.g., 3G, LTE)    -   PC-TCP concurrently over multiple wireless base stations (e.g.,        via multiple wireless LAN access points)

In some examples, Network Coding is used such that the multiple pathsfrom a server node to a client node pass through one or moreintermediate nodes at which the data is recoded, thereby causinginformation for different data units to effectively traverse differentpaths through the network.

One motivation for multipath connection between a pair of endpointsaddresses possible preferential treatment of TCP traffic rather than UDPtraffic. Some networks (e.g. certain public Wi-Fi, cable televisionnetworks, etc.) may limit the rate of UDP traffic, or drop UDP packetspreferentially compared to TCP (e.g., in the case of congestion). It maybe desirable to be able to detect such scenarios efficiently withoutlosing performance. In some embodiments, a PC-TCP session initiallyestablishes and divides the transmitted data across both a TCP and a UDPconnection. This allows comparison of the throughput achieved by bothconnections while sending distinct useful data on each connection. Anidentifier is included in the initial TCP and UDP handshake packets toidentify the two connections as belonging to the same coded PC-TCPsession, and non-blocking connection establishment can be employed so asto allow both connections to be opened at the outset without additionaldelay. The transmitted data is divided across the two connections usinge.g. round-robin (sending alternating packets or runs of packets on eachconnection) or load-balancing/back pressure scheduling (sending eachpacket to the connection with the shorter outgoing data queue). Suchalternation or load balancing can be employed in conjunction withtechniques for dealing with packet reordering. Pacing rate andcongestion window size can be controller separately for the UDP and theTCP connection, or can be controlled together. By controlling the twoconnections together (e.g., using only a single congestion window toregulate the sum of the number of packets in flight on both the TCP andUDP connections) may provide a greater degree of “fairness” as comparedto separate control.

In some examples, the adjustment of the fraction of messages transmittedover each data path/protocol is determined according to the relativeperformance/throughput of the data paths/protocols. In some examples,the adjustment of allocation of messages occurs only during an initialportion of the transmission. In other examples, the adjustment ofallocation of messages occurs on an ongoing basis throughout thetransmission. In some examples, the adjustment reverses direction (e.g.,when a data path stops preferentially dropping UDP messages, the numberof messages transmitted over that data path may increase).

In some examples, the adjustment of the fraction of messages transmittedover each data path/protocol is determined for one data communicationsession/connection and is re-used for a number of subsequent datacommunication sessions/connections. A re-test interval may be used tospecify the number of subsequent sessions/connections that can passwithout retesting and adjusting the fraction of messages, since aperformance hit is associated with re-testing. In some examples, aninitial fraction of messages transmitted over each data path/protocol isdetermined based on a probability of a client running on a throttlednetwork.

In some embodiments, the PC-TCP maintains both the UDP based traffic andthe TCP based traffic for the duration of the session. In otherembodiments, the PC-TCP module compares the behavior of the UDP and TCPtraffic, for example over a period specified in terms of time intervalor number of packets, where these quantities specifying the period canbe set as configuration parameters and/or modified based on previouscoded TCP sessions, e.g. the comparison period can be reduced oreliminated if information on relative TCP/UDP performance is availablefrom recent PC-TCP sessions. If the UDP connection achieves betterthroughput, the PC-TCP session can shift to using UDP only. If the TCPconnection achieves better throughput, the PC-TCP session can shift tousing TCP. In some embodiments, different types of traffic are sent overthe TCP link rather than the UDP link. In one such example, the UDPconnection is used to send some forward error correction for packetswhere it is beneficial to reduce retransmission delays, e.g. the lastblock of a file or intermediate blocks of a stream. In this example, theuncoded packets may be sent over a TCP stream with forward errorcorrection packets sent over UDP. If the receiver can use the forwarderror correction packets to recover from erasures in the TCP stream, amodified implementation of the TCP component of the receiver's PC-TCPmodule may be able to avoid using a TCP-based error recovery procedure.On the other hand, non-delivery of a forward error correction packetdoes not cause an erasure of the data that is to be recovered at thereceiver, and therefore unless there is an erasure both on the UDP pathand on the TCP path, dropping of a UDP packet does not cause delay.

2.5.2 Distributed Source

In some examples, multiple server nodes communicate with a client node.One way this can be implemented is with there being multiplecommunication sessions each involving one server node and one clientnode. In such an implementation, there is little or no interactionbetween a communication session between one server node and the clientnode and another communication session between another server node andthe client node. In some examples, each server node may have differentparts of a multimedia file, with each server providing its parts forcombination at the client node.

2.5.3 Distributed Content Delivery

In some examples, there is some relationship between the contentprovided by different servers to the client. One example of such arelationship is use of a distributed RAID approach in which redundancyinformation (e.g., parity information) for data units at one or moreservers is stored at and provided from another server. In this way,should a data unit not reach the client node from one of the servernodes, the redundancy information may be preemptively sent or requestedfrom the other node, and the missing data unit reconstructed.

In some examples, random linear coding is performed on data units beforethey are distributed to multiple server nodes as an alternative to useof distributed RAID. Then each server node establishes a separatecommunication session with the client node for delivery of part of thecoded information. In some of these examples, the server nodes havecontent that has already been at least partially encoded and thencached, thereby avoiding the necessity of repeating that partialencoding for different client nodes that will receive the sameapplication data units. In some examples, the server nodes may implementsome of the functionality of the PC modules for execution duringcommunication sessions with client nodes, for example, having theability to encode further redundancy information in response toacknowledgment information (i.e., negative acknowledgment information)received from a client node.

In some implementations, the multiple server nodes are content deliverynodes to which content is distributed using any of a variety of knowntechniques. In other implementations, these multiple server nodes areintermediary nodes at which content from previous content deliverysessions was cached and therefore available without requiringre-delivery of the content from the ultimate server node.

In some examples of distributed content delivery, each server to clientconnection is substantially independent, for example, with independentlydetermined communication parameters (e.g., error correction parameters,congestion window size, pacing rate, etc.). In other examples, at leastsome of the parameters are related, for example, with characteristicsdetermined on one server-to-client connection being used to determinehow the client node communicates with other server nodes. For example,packet arrival rate, loss rate, and differences in one-way transmissionrate, may be measured on one connections and these parameters may beused in optimizing multipath delivery of data involving other servernodes. One manner of optimization may involve load balancing acrossmultiple server nodes or over communication links on the paths from theserver nodes to the client nodes.

In some implementations, content delivery from distributed server nodesmaking use of PC-TCP, either using independent sessions or usingcoordination between sessions, may achieve the performance ofconventional distributed content delivery but requiring a smaller numberof server nodes. This advantage may arise due to PC-TCP providing lowerlatency and/or lower loss rates than achieved with conventional TCP.

2.6 Multicast

FIGS. 19-20 show two examples of delivery of common content to multipledestination nodes simultaneously via multicast connections. Theadvantage of multicast is that a single packet or block of N packets hasto be sent by the source node into the network and the network willattempt to deliver the packets to all destination nodes in the multicastgroup. If the content needs to be delivered reliably, then TCP will mostlikely be used as the transport layer protocol. To achieve reliability,TCP requires destination nodes to respond with acknowledgments andspecify the packets that each destination node is missing. If there are10s of thousands or 100s of thousands of receivers, and each destinationnode is missing a different packet or set of packets, the number ofdifferent retransmissions to the various receivers will undercut theadvantages of the simultaneous transmission of the content to alldestination nodes at once. With network coding and forward errorcorrection, a block of N packets can be sent to a large number ofmulticast destination nodes at the same time. The paths to thesemultiple destination nodes can be similar (all over a large WiFi orEthernet local area network) or disparate (some over WiFi, some overcellular, some over fiber links, and some over various types ofsatellite networks). The algorithms described above that embodytransmission and congestion control, forward error correction, senderbased pacing, receiver based pacing, stream based parameter tuning,detection and correction for missing and out of order packets, use ofinformation across multiple connections, fast connection start and stop,TCP/UDP fallback, cascaded coding, recoding by intermediate nodes, andcoding of the ACKs can be employed to improve the throughput andreliability of delivery to each of the multicast destination node. Whenlosses are detected and coding is used, the extra coded packets can besent to some or all destination nodes. As long as N packets are receivedat each destination node, the missing packets at each destination nodecan be reconstructed from the coded packets if the number of extra codedpackets match or exceed the number of packets lost at all of thereceivers. If fewer than N packets are received at any of thedestination nodes, any set of different coded packets from the block ofN packets can be retransmitted and used to reconstruct any missingpacket in the block at each of the destination nodes. If somedestination nodes are missing more than one packet, then the maximumnumber of coded packets to be retransmitted will be equal to the largestnumber of packets that are missing by any of the destination nodes.These few different coded packets can be used to reconstruct the missingpackets at each of the destination nodes. For example, if the mostpackets missing at any destination node are four, then any fourdifferent coded packets can be retransmitted.

2.7 Further Illustrative Examples

FIGS. 21A-21K show exemplary embodiments of data communication systemsand devices and highlight various ways to implement the novel PC-TCPdescribed herein. These configurations identify some of the possiblenetwork devices, configurations, and applications that may benefit fromusing PC-TCP, but there are many more devices, configurations andapplications that may also benefit from PC-TCP. The followingembodiments are described by way of example, not limitation.

In an exemplary embodiment depicted in FIG. 21A, a user device 404 suchas a smartphone, a tablet, a computer, a television, a display, anappliance, a vehicle, a home server, a gaming console, a streaming mediabox and the like, may include a PC-TCP proxy that may interface withapplications running in the user device 404. The application on the userdevice 404 may communicate with a resource in the cloud 402 a such as aserver 408. The server 408 may be a file server, a web server, a videoserver, a content server, an application server, a collaboration server,an FTP server, a list server, a telnet server, a mail server, a proxyserver, a database server, a game server, a sound server, a printserver, an open source server, a virtual server, an edge server, astorage device and the like, and may include a PC-TCP proxy that mayinterface with applications and/or processes running on the server 408.In embodiments, the server in the cloud may terminate the PC-TCPconnection and interface with an application on the server 408 and/ormay forward the data on to another electronic device in the network. Inembodiments, the data connection may travel a path that utilizes theresources on a number of networks 402 a, 402 b. In embodiments, PC-TCPmay be configured to support multipath communication such as for examplefrom a video server 408 through a peering point 406, though a carriernetwork 402 b, to a wireless router or access point 410 to a user device404 and from a video server 408 through a peering point 406, though acarrier network 402 b, to a cellular base station or cell transmitter412 to a user device 404. In embodiments, the PC-TCP may includeadjustable parameters that may be adjusted to improve multipathperformance. In some instances, the exemplary embodiment shown in FIG.21A may be referred to as an over-the-top (OTT) embodiment.

In embodiments, such as the exemplary embodiments shown in FIG. 21B andFIG. 21C, other devices in the network may comprise PC-TCP proxies. Forexample, the wireless access point or router 410 and the base station orcell transmitter 412 may comprise PC-TCP proxies. In embodiments, theuser device 404 may also comprise a PC-TCP proxy (FIG. 21C) or it maynot (FIG. 21B). If the user device does not comprise a PC-TCP proxy, itmay communicate with the access point 410 and/or base station 412 usinga wireless or cellular protocol and/or conventional TCP or UDP protocol.The PC-TCP proxy in either or both the access point 410 and base station412 may receive data packets using these conventional communications andmay convert these communications to the PC-TCP for a connection to videoserver 408. In embodiments, if conventional TCP provides the highestspeed connection between the end user device 404 and/or the access point410 or the base station 412, then the PC-TCP proxy may utilize only someor all of the features in PC-TCP that may be compliant with and maycomplement conventional TCP implementations and transmit the data usingthe TCP layer.

FIG. 21D shows an exemplary embodiment where a user device may comprisea PC-TCP proxy and may communicate with a PC-TCP proxy server 408 on aninternet. In this embodiment, an entity may provide support for highspeed internet connections by renting, buying services from, ordeploying at least one server in the network and allowing other serversor end user devices to communicate with it using PC-TCP. The at leastone server in the network running PC-TCP may connect to other resourcesin the network and/or end users using TCP or UDP.

In embodiments, such as the exemplary embodiments shown in FIG. 21E andFIG. 21F, other devices in the network may comprise PC-TCP proxies. Forexample, the wireless access point or router 410 and the base station orcell transmitter 412 may comprise PC-TCP proxies. In embodiments, theuser device 404 may also comprise a PC-TCP proxy (FIG. 21F) or it maynot (FIG. 21E). If the user device does not comprise a PC-TCP proxy, itmay communicate with the access point 410 and/or base station 412 usinga wireless or cellular protocol and/or conventional TCP or UDP protocol.The PC-TCP proxy in either or both the access point 410 and base station412 may receive data packets using these conventional communications andmay convert these communications to the PC-TCP for a connection toPC-TCP server 408. In embodiments, if conventional TCP provides thehighest speed connection between the end user device 404 and/or theaccess point 410 or the base station 412, then the PC-TCP proxy mayutilize only some or all of the features in PC-TCP that may be compliantwith and may complement conventional TCP implementations and transmitthe data using the TCP layer.

In embodiments, at least some network servers 408 may comprise PC-TCPproxies and may communicate with any PC-TCP servers or devices usingPC-TCP. In other embodiments, network servers may communicate withPC-TCP servers or devices using conventional TCP and/or other transportprotocols running over UDP.

In exemplary embodiments as depicted in FIG. 21G, ISPs and/or carriersmay host content on one or more servers that comprise PC-TCP proxies. Inembodiments, devices such as set-top boxes, cable boxes, digital videorecorders (DVRs), modems, televisions, smart televisions, internettelevisions, displays, and the like may comprise PC-TCP proxies. A userdevice 404 such as described above, may include a PC-TCP proxy that mayinterface with applications running in the user device 404. Theapplication on the user device 404 may communicate with a resource inthe cloud 402 c such as a server 408. The server 408 may be any type ofcommunications server as describe above, and may include a PC-TCP proxythat may interface with applications and/or processes running on theserver 408. In embodiments, the server in the cloud may terminate thePC-TCP connection and interface with an application on the server 408and/or may forward the data on to another electronic device in thenetwork. In embodiments, the data connection may travel a path thatutilizes the resources on a number of networks 402 a, 402 b, 402 c. Inembodiments PC-TCP may be configured to support multipath communicationsuch as for example from a video server 408 through a direct peeringpoint (DP) 406, to a wireless router or access point 410 or a basestation 412 to a user device 404 and from a video server 408 directly toan access point 410 and/or to a cellular base station or celltransmitter 412 to a user device 404. In embodiments, the PC-TCP mayinclude adjustable parameters that may be adjusted to improve multipathperformance.

The exemplary placements of networking devices in the communicationscenarios described above should not be taken as limitations. It shouldbe recognized that PC-TCP proxies can be placed in any network deviceand may support any type of data connection. That is, any type ofend-user device, switching device, routing device, storage device,processing device and the like, may comprise PC-TCP proxies. Also,PC-TCP proxies may reside only in the end-nodes of a communication pathand/or only at two nodes along a connection path. However, PC-TCPproxies may also reside in more than two nodes of a communication pathand may support multi-cast communications and multipath communications.PC-TCP proxies may be utilized in point-to-point communication networks,multi-hop networks, meshed networks, broadcast networks, storagenetworks, and the like.

3 Packet Coding (PC)

The description above focuses on architectures in which a packet codingapproach is deployed, and in particular architectures in which atransport layer PC-TCP approach is used. In the description below, anumber of features of PC-TCP are described. It should be understood thatin general, unless otherwise indicated, these features are compatiblewith one another and can be combined in various combinations to addressparticular applications and situations.

3.1 Data Characteristics

As introduced above, data units (e.g., audio and/or video frames) aregenerally used to form data packets, for example, with one data unit perdata packet, with multiple data units per data packet, or in someinstances separating individual data units into multiple data packets.In some applications, the data units and associated data frames form astream (e.g., a substantially continuous sequence made available overtime without necessarily having groupings or boundaries in thesequence), while in other applications, the data units and associateddata frames form one or more batches (e.g., a grouping of data that isrequired as a whole by the recipient).

In general, stream data is generated over time at a source and consumedat a destination, typically at a substantially steady rate. An exampleof a stream is a multimedia stream associated with person-to-personcommunication (e.g., a multimedia conference). Delay (also referred toas latency) and variability in delay (also referred to as jitter) areimportant characteristics of the communication of data units from asource to a destination.

An extreme example of a batch is delivery of an entire group of data,for example, a multiple gigabyte sized file. In some such examples,reducing the overall time to complete delivery (e.g., by maximizingthroughput) of the batch is of primary importance. One example of batchdelivery that may have very sensitive time (and real-time update)restraints is database replication.

In some applications, the data forms a series of batches that requiredelivery from a source to a destination. Although delay in start ofdelivery and/or completion of delivery of a batch of data units may beimportant, in many applications overall throughput may be mostimportant. An example of batch delivery includes delivery of portions ofmultimedia content, for instance, with each batch corresponding tosections of viewing time (e.g., 2 seconds of viewing time or 2 MB perbatch), with content being delivered in batches to the destination wherethe data units in the batches are buffered and used to construct acontinuous presentation of the content. As a result, an importantconsideration is the delivery of the batches in a manner that providescontinuity between batches for presentation, without “starving” thedestination application because a required batch has not arrived intime. In practice, such starving may cause “freezing” of videopresentation in multimedia, which is a phenomenon that is all toofamiliar to today's users of online multimedia delivery. Anotherimportant consideration is reduction in the initial delay in providingthe data units of the first batch to the destination application. Suchdelay is manifested, for example, in a user having to wait for initialstartup of video presentation after selecting multimedia for onlinedelivery. Another consideration in some applications is overallthroughput. This may arise, for example, if the source application hascontrol over a data rate of the data units, for example, being able toprovide a higher fidelity version of the multimedia content if higherthroughput can be achieved. Therefore, an important consideration may beproviding a sufficiently high throughput in order to enable delivery ofa high fidelity version of the content (e.g., as opposed to a greatlycompressed version or a backed-off rate of the content resulting inlower fidelity).

Various packet coding approaches described below, or selection ofconfiguration parameters of those approaches, address considerationsthat are particularly relevant to the nature of the characteristics ofthe data being transported. In some examples, different approaches orparameters are set in a single system based on a runtime determinationof the nature of the characteristics of the data being transported.

3.2 Channel Characteristics

In general, the communication paths that link PC-TCP source anddestination endpoints exhibit both relatively stationary or consistentchannel characteristics, as well as transient characteristics.Relatively stationary or consistent channel characteristics can include,for example, capacity (e.g., maximum usable throughput), latency (e.g.,transit time of packets from source to destination, variability intransit time), error rate (e.g., average packet erasure or error rate,burst characteristics of erasures/errors). In general, such relativelystationary or consistent characteristics may depend on the nature of thepath, and more particularly on one or more of the links on the path. Forexample, a path with a link passing over a 4G cellular channel mayexhibit very different characteristics than a path that passes over acable television channel and/or a WiFi link in a home. As discussedfurther below, at least some of the approaches to packet coding attemptto address channel characteristic differences between types ofcommunication paths. Furthermore, at least some of the approachesinclude aspects that track relatively slow variation in characteristics,for example, adapting to changes in average throughput, latency, etc.

Communication characteristics along a path may also exhibit substantialtransient characteristics. Conventional communication techniques includeaspects that address transient characteristics resulting from congestionalong a communication path. It is well known that as congestionincreases, for example at a node along a communication path, it isimportant that traffic is reduced at that node in order to avoid anunstable situation, for instance, with high packet loss resulting frombuffer overruns, which then further increases data rates due toretransmission approaches. One common approach to addressingcongestion-based transients uses an adaptive window size of “in flight”packets that have not yet been acknowledged by their destinations. Thesize of the window is adapted at each of the sources to avoidcongestion-based instability, for example, by significantly reducing thesize of the window upon detection of increased packet erasure rates.

In addressing communication over a variety of channels, it has beenobserved that transients in communication characteristics may not be duesolely to conventional congestion effects, and that conventionalcongestion avoidance approaches may not be optimal or even desirable.Some effects that may affect communication characteristics, and thatmay, therefore, warrant adaptation of the manner in which data istransmitted can include one or more of the following:

-   -   Effects resulting from cell handoff in cellular systems,        including interruptions in delivery of packets or substantial        reordering of packets delivered after handoff;    -   Effects resulting from “half-duplex” characteristics of certain        wireless channels, for example, in WiFi channels in which return        packets from a destination may be delayed until the wireless        channel is acquired for upstream (i.e., portable device to        access point) communication;    -   Effects of explicit data shaping devices, for example, intended        to throttle certain classes of communication, for instance,        based on a service provider's belief that that class of        communication is malicious or is consuming more than a fair        share of resources.

Although transient effects, which may not be based solely on congestion,may be tolerated using conventional congestion avoidance techniques, oneor more of the approaches described below are particularly tailored tosuch classes of effects with the goal of maintaining efficient use of achannel without undue “over-reaction” upon detection of a transientsituation, while still avoiding causing congestion-based packet loss.

3.3 Inter-Packet Coding

In general, the coding approaches used in embodiments described in thisdocument make use of inter-packet coding in which redundancy informationis sent over the channel such that the redundancy information in onepacket is generally dependent on a set of other packets that have beenor will be sent over the channel. Typically, for a set of N packets ofinformation, a total of N+K packets are sent in a manner that erasure orany K of the packets allows reconstruction of the original N packets ofinformation. In general, a group of N information packets, or a group ofN+K packets including redundancy information (depending on context), isreferred to below as a “block” or a “coding block”. One example of sucha coding includes N information packets without further coding, and thenK redundancy packets, each of which depends on the N informationpackets. However, it should be understood more than K of the packets(e.g., each of the N+K packets) may in some embodiments depend on allthe N information packets.

3.3.1 Forward Error Correction and Repair Retransmission

Inter-packet coding in various embodiments described in this documentuse one or both of pre-emptive transmission of redundant packets,generally referred to as forward error correction (FEC), andtransmission of redundant packets upon an indication that packets haveor have a high probability of having been erased based on feedback,which is referred to below as repair and/or retransmission. The feedbackfor repair retransmission generally comes from the receiver, but moregenerally may come from a node or other channel element on the path tothe receiver, or some network element having information about thedelivery of packets along the path. In the FEC mode, K redundant packetsmay be transmitted in order to be tolerant of up to K erasures of the Npackets, while in the repair mode, in some examples, for each packetthat the transmitter believes has been or has high probability of havingbeen erased, a redundant packet is transmitted from the transmitter,such that if in a block of N packets, K packets are believed to havebeen erased based on feedback, the transmitter sends at least anadditional K packets.

As discussed more fully below, use of a forward error correction modeversus a repair mode represents a tradeoff between use of more channelcapacity for forward error correction (i.e., reduced throughout ofinformation) versus incurring greater latency in the presence oferasures for repair retransmission. As introduced above, the datacharacteristics being transmitted may determine the relative importanceof throughput versus latency, and the PC-TCP modules may be configuredor adapted accordingly.

If on average the packet erasure rate E is less than K/(N+K), then “onaverage” the N+K packets will experience erasure of K or fewer of thepackets and the remaining packets will be sufficient to reconstruct theoriginal N. Of course even if E is not greater than K/(N+K), randomvariability, non-stationarity of the pattern of erasures, etc. resultsin some fraction of the sets of N+K packets having greater than Kerasures, so that there is insufficient information to reconstruct the Npackets at the destination. Therefore, even using FEC, at least somegroups of N information packets will not be reconstructable. Note, forexample, with E=0.2, N=8, and K=2, even though only 2 erasures may beexpected on average, the probability of more than 2 erasures is greaterthan 30%, and even with E=0.1 this probability is greater than 7%,therefore the nature (e.g., timing, triggering conditions etc.) of theretransmission approaches may be significant, as discussed furtherbelow. Also as discussed below, the size of the set of packets that arecoded together is significant. For example, increasing N by a factor of10 to K+N=100 reduces the probability of more than the average number of20 erasures (i.e., too many erasures to reconstruct the N=80 datapackets) from over 7% to less than 0.1%.

Also as discussed further below, there is a tradeoff between use oflarge blocks of packets (i.e., large N) versus smaller blocks. For aparticular code rate R=N/(N+K), longer blocks yield a higher probabilityof being able to fully recover the N information packets in the presenceof random errors. Accordingly, depending on the data characteristics,the PC-TCP modules may be configured to adapt to achieve a desiredtradeoff.

In general, in embodiments that guarantee delivery of the N packets,whether or not FEC is used, repair retransmission approaches are used toprovide further information for reconstructing the N packets. Ingeneral, in preferred embodiments, the redundancy information is formedin such a manner that upon an erasure of a packet, the redundancyinformation that is sent from the transmitter does not depend on thespecific packets that were erased, and is nevertheless suitable forrepairing the erasure independent of which packet was erased.

3.3.2 Random Linear Coding

In general, a preferred approach to inter-packet coding is based onRandom Linear Network Coding (RLNC) techniques. However, it should beunderstood that although based on this technology, not all features thatmay be associated with this term are necessarily incorporated. Inparticular, as described above in the absence of intermediate nodes thatperform recoding, there is not necessarily a “network” aspect to theapproach. Rather, redundancy information is generally formed bycombining the information packets into coded packets using arithmeticcombinations, and more specifically, as sums of products of coefficientsand representation of the information packets over arithmetic fields,such as finite fields (e.g., Galois Fields of order p^(n)). In general,the code coefficients are chosen from a sufficiently large finite fieldin a random or pseudo-random manner, or in another way that thecombinations of packets have a very low probability or frequency ofbeing linearly dependent. The code coefficients, or a compressed version(e.g., as a reference into a table shared by the transmitter andreceiver), are included in each transmitted combination of data units(or otherwise communicated to the receiver) and used for decoding at thereceiver. Very generally, the original information packets may berecovered at a receiver by inverting the arithmetic combinations. Forexample, a version of Gaussian Elimination may be used to reconstructthe original packets from the coded combinations. A key feature of thisapproach is that for a set of N information packets, as soon as thereceiver has at least N linearly independent combinations of thoseinformation packets in received packets, it can reconstruct the originaldata units. The term “degree of freedom” is generally used below torefer to a number of independent linear combinations, such that if Ndegrees of freedom have been specified for N original packets, then theN original packets can be reconstructed; while if fewer than N degreesof freedom are available, it may not be possible to fully reconstructany of the N original packets. If N+K linearly independent linearcombinations are sent, then any N received combinations (i.e., Nreceived degrees of freedom) are sufficient to reconstruct the originalinformation packets.

In some examples, the N+K linearly independent combinations comprise Nselections of the N “uncoded” information packets (essentially N−1 zerocoefficients and one unit coefficient for each uncoded packet), and Kcoded packets comprising the random arithmetic combination with Nnon-zero coefficients for the N information packets. The N uncodedpackets are transmitted first, so that in the absence of erasures theyshould be completely received as soon as possible. In the case of oneerasure of the original N packets, the receiver must wait for thearrival of one redundant packet (in addition to the N−1 originalpackets), and once that packet has arrived, the erased packet may bereconstructed. In the case of forward error correction, the K redundantpackets follow (e.g., immediately after) the information packets, andthe delay incurred in reconstructing the erased information packetdepends on the transmission time of packets. In the case of repairretransmission, upon detection of an erasure or high probability of anerasure, the receiver provides feedback to the transmitter, which sendsthe redundancy information upon receiving the feedback. Therefore, thedelay in being able to reconstruct the erased packet depends on theround-trip-time from the receiver to the transmitter and back.

As discussed in more detail below, feedback from the receiver to thetransmitter may be in the form of acknowledgments sent from the receiverto the transmitter. This feedback in acknowledgments at least informsthe transmitter of a number of the N+K packets of a block that have beensuccessfully received (i.e., the number of received degrees of freedom),and may provide further information that depends on the specific packetsthat have been received at the receiver although such furtherinformation is not essential.

As introduced above, packets that include the combinations of originalpackets generally also include information needed to determine thecoefficients used to combine the original packets, and informationneeded to identify which original packets were used in the combination(unless this set, such as all the packets of a block, is implicit). Insome implementations, the coefficients are explicitly represented in thecoded packets. In some embodiments, the coefficients are encoded withreference to shared information at the transmitter and the receiver. Forinstance, tables of pre-generated (e.g., random, pseudo random, orotherwise selected) coefficients, or sets of coefficients, may be storedand references into those tables are used to determine the values of thecoefficients. The size of such a table determines the number of paritypackets that can be generated while maintaining the linear independenceof the sets of coefficients. It should be understood that yet other waysmay be used to determine the coefficients.

Another feature of random linear codes is that packets formed as linearcombinations of data units may themselves be additively combined toyield combined linear combinations of data units. This process isreferred to in some instances as “recoding”, as distinct from decodingand then repeating encoding.

There are alternatives to the use of RLNC, which do not necessarilyachieve similar optimal (or provably optimum, or near optimal)throughput as RLNC, but that give excellent performance in somescenarios when implemented as described herein. For example, variousforms of parity check codes can be used. Therefore, it should beunderstood that RLNC, or any particular aspect of RLNC, is not anessential feature of all embodiments described in this document.

3.4 Batch Transmission

As introduced above, in at least some applications, data to betransmitted from a transmitter to a receiver forms a batch (i.e., asopposed to a continuous stream), with an example of a batch being a fileor a segment (e.g., a two second segment of multimedia) of a file.

In an embodiment of the PC-TCP modules, the batch is transferred fromthe transmitter to the receiver as a series of blocks, with each blockbeing formed from a series of information packets. In general, eachblock has the same number of information packets, however, use of samesize blocks is not essential.

The transmitter PC-TCP module generally receives the data units from thesource application and forms the information packets of the successiveblocks of the batch. These information packets are queued at thetransmitter and transmitted on the channel to the receiver. In general,at the transmitter, the dequeueing and transmission of packets to thereceiver makes use of congestion control and/or rate control mechanismsdescribed in more detail below. The transmitter PC-TCP also retains theinformation packets (or sufficient equivalent information) to constructredundancy information for the blocks. For instance, the transmitterPC-TCP buffers the information packets for each block for which thereremains the possibility of an unrecovered erasure of a packet duringtransit from the transmitter to the receiver.

In general, the receiver provides feedback to the transmitter. Variousapproaches to determining when to provide the feedback and whatinformation to provide with the feedback are described further below.The feedback provides the transmitter with sufficient information todetermine that a block has been successfully received and/orreconstructed at the receiver. When such success feedback for a blockhas been received, the transmitter no longer needs to retain theinformation packets for the block because there is no longer thepossibility that redundancy information for the block will need to besent to the receiver.

The feedback from the receiver to the transmitter may also indicate thata packet is missing. Although in some cases the indication that a packetis missing is a premature indication of an erasure, in this embodimentthe transmitter uses this missing feedback to trigger sending redundantinformation for a block. In some examples, the packets for a block arenumbered in sequence of transmission, and the feedback represents thehighest number received and the number of packets (i.e., the number ofdegrees of freedom) received (or equivalently the number of missingpackets or remaining degrees of freedom needed) for the block. Thetransmitter addresses missing packet feedback for a block through thetransmission of redundant repair blocks, which may be used by thereceiver to reconstruct the missing packets and/or original packets ofthe block.

As introduced above, for each block, the transmitter maintainssufficient information to determine the highest index of a packetreceived at the receiver, the number of missing packets transmittedprior to that packet, and the number of original or redundancy packetsafter the highest index received that have been transmitted (i.e., are“in flight” unless erased in transit) or queued for transmission at thetransmitter.

When the transmitter receives missing packet feedback for a block, ifthe number of packets for the block that are “in flight” or queue wouldnot be sufficient if received successfully (or are not expected to be inview of the erasure rate), the transmitter computes (or retrievesprecomputed) a new redundant packet for the block and queues it fortransmission. Such redundancy packets are referred to as repair packets.In order to reduce the delay in reconstructing a block of packets at thereceiver, the repair packets are sent preferentially to the informationpackets for later blocks. For instance, the repair packets are queued ina separate higher-priority queue that is used to ensure transmission ofrepair packets preferentially to the queue of information packets.

In some situations, feedback from the receiver may have indicated that apacket is missing. However, that packet may later arrive out of order,and therefore a redundant packet for that block that was earliercomputed and queued for transmission is no longer required to bedelivered to the receiver. If that redundant packet has not yet beentransmitted (i.e., it is still queued), that packet may be removed fromthe queue thereby avoiding wasted use of channel capacity for a packetthat will not serve to pass new information to the receiver.

In the approach described above, redundancy packets are sent as repairpackets in response to feedback from the receiver. In some examples,some redundancy packets are sent preemptively (i.e., as forward errorcorrection) in order to address possible packet erasures. One approachto send such forward error correction packets for each block. However,if feedback has already been received at the transmitter that asufficient number of original and/or coded packets for a block have beenreceived, then there is no need to send further redundant packets forthe block.

In an implementation of this approach, the original packets for all theblocks of the batch are sent first, while repair packets are beingpreferentially sent based on feedback from the receiver. After all theoriginal packets have been transmitted, and the queue of repair packetsis empty, the transmitter computes (or retrieves precomputed) redundancypackets for blocks for which the transmitter has not yet receivedfeedback that the blocks have been successfully received, and queuesthose blocks as forward error correction packets for transmission in thefirst queue. In general, because the repair blocks are sent with higherpriority than the original packets, the blocks for which successfeedback has not yet been received are the later blocks in the batch(e.g., a trailing sequence of blocks of the batch).

In various versions of this approach, the number and order oftransmission of the forward error correction packets are determined invarious ways. A first way uses the erasure rate to determine how manyredundant packets to transmit. One approach is to send at least oneredundant packet for each outstanding block. Another approach is to senda number of redundancy packets for each outstanding block so that basedon an expectation of the erasure rate of the packets that are queued andin flight for the block will yield a sufficient number of successfullyreceived packets in order to reconstruct the block. For example, if afurther n packets are needed to reconstruct a block (e.g., a number n<Npackets of the original N packets with N−n packets having been erased),then n+k packets are sent, for instance, with n+k≥n/E, where E is anestimate of the erasure rate on the channel.

Another way of determining the number and order of forward errorcorrection packets addresses the situation in which a block transmissiontime is substantially less than the round-trip-time for the channel.Therefore, the earliest of the blocks for which the transmitter has notreceived success feedback may, in fact, have the success feedback inflight from the receiver to the transmitter, and therefore sendingforward error correction packets may be wasteful. Similarly, even iffeedback indicating missing packet feedback for a block is receivedsufficiently early, the transmitter may still send a repair packetwithout incurring more delay in complete reconstruction of the entirebatch than would be achieved by forward error correction.

In an example, the number of forward error correction packets queued foreach block is greater for later blocks in the batch than for earlierones. A motivation for this can be understood by considering the lastblock of the batch where it should be evident that it is desirable tosend a sufficient number of forward error correction packets to ensurehigh probability of the receiver having sufficient information toreconstruct the block without the need from transmission of a repairpacket and the associated increase in latency. On the other hand, it ispreferable to send fewer forward error correction packets for theprevious (or earlier) block because, in the face of missing packetfeedback from the receiver, the transmitter may be able to send a repairpacket before forward error correction packets for all the later blockshave been sent, thereby not incurring a delay in overall delivery of thebatch.

In one implementation, after all the original packets have been sent,and the transmitter is in the forward error correction phase in which itcomputes and sends the forward error correction packets, if thetransmitter receives a missing packet feedback from the receiver, itcomputes and sends a repair packet for the block in question (ifnecessary) as described above, and clears the entire queue of forwarderror correction packets. After the repair packet queue is again empty,the transmitter again computes and queues forward error correctionpackets for the blocks for which it has not yet received successfeedback. In an alternative somewhat equivalent implementation, ratherthan clearing the forward error correction queue upon receipt of amissing packet feedback, the transmitter removes forward errorcorrection packets from the queue as they are no longer needed based onfeedback from the receiver. In some examples, if success feedback isreceived for a block for which there are queued forward error correctionpackets, those forward error correction packets are removed from thequeue. In some examples, the feedback from the receiver may indicatethat some but not all of the forward error correction packets in thequeue are no longer needed, for example, because out-of-order packetswere received but at least some of the original packets are stillmissing.

An example of the way the transmitter determines how many forward errorcorrection packets to send is that the transmitter performs acomputation:

(N+g(i)−a _(i))/(1−p)−f _(i)

where

p=smoothed loss rate,

N=block size,

i=block index defined as number of blocks from last block,

a_(i)=number of packets acked from block i,

f_(i)=packets in-flight from block i, and

g(i)=a decreasing function of i,

to determine the number of FEC packets for a block.

In some examples, g(i) is determined as a maximum of a configurableparameter, m and N-i. In some examples, g(i) is determined as N−p(i)where p is a polynomial, with integer rounding as needed.

It should be understood that in some alternative implementations, atleast some forward error correction packets may be interspersed with theoriginal packets. For example, if the erasure rate for the channel isrelatively high, then at least some number of redundancy packets may beneeded with relatively high probability for each block, and there is anoverall advantage to preemptively sending redundant FEC packets as soonas possible, in addition to providing the mechanism for feedback basedrepair that is described above.

It should be also understood that use of subdivision of a batch intoblocks is not necessarily required in order to achieve the goal ofminimizing the time to complete reconstruction of the block at thereceiver. However, if the forward error correction is applied uniformlyto all the packets of the batch, then the preferential protection oflater packets would be absent, and therefore, latency caused by erasureof later packets may be greater than using the approach described above.However, alternative approaches to non-uniform forward error protection(i.e., introduction of forward error correction redundancy packets) maybe used. For example, in the block based approach described above,packets of the later blocks each contribute to a greater number offorward error correction packets than do earlier ones, and analternative approach to achieving this characteristic maybe to use anon-block based criterion to construction of the redundancy packets inthe forward error correction phase. However, the block based approachdescribed above has advantages of relative simplicity and generalrobustness, and therefore even if marginally “suboptimal” provides anoverall advantageous technical solution to minimizing the time tocomplete reconstruction within the constraint of throughput and erasureon the channel linking the transmitter and receiver.

Another advantage of using a block-based approach is that, for example,when a block within the batch, say the m^(th) block of M blocks of thebatch has an erasure, the repair packet that is sent from thetransmitter depends only on the N original packets of the m^(th) block.Therefore, as soon as the repair packet arrives, and the available(i.e., not erased)N−1 packets of the block arrive, the receiver has theinformation necessary to repair the block. Therefore, by constructingthe repair packet without contribution of packets in later blocks of thebatch, the latency of the reconstruction of the block is reduced.Furthermore, by having the repair packets depend on only N originalpackets, the computation required to reconstruct the packets of theblock is less than if the repair packets depend on more packets.

It should be understood that even in the block based transmission of abatch of packets, the blocks are not necessarily uniform in size, andare not necessarily disjoint. For example, blocks may overlap (e.g., by50%, 75%, etc.) thereby maintaining at least some of the advantages ofreduced complexity in reconstruction and reduced buffering requirementsas compared to treating the batch as one block. An advantage of suchoverlapping blocks may be a reduced latency in reconstruction becauserepair packets may be sent that do not require waiting for originalpackets at the receiver prior to reconstruction. Furthermore,non-uniform blocks may be beneficial, for example, to increase theeffectiveness of forward error correction for later block in a batch byusing longer blocks near the end of a batch as compared to near thebeginning of a batch.

In applications in which the entire batch is needed by the destinationapplication before use, low latency of reconstruction may be desirableto reduce buffering requirements in the PC-TCP module at the receiver(and at the transmitter). For example, all packets that may contributeto a later received repair packet are buffered for their potentialfuture use. In the block based approach, once a block is fullyreconstructed, then the PC-TCP module can deliver and discard thosepackets because they will not affect future packet reconstruction.

Although described as an approach to delivery of a batch of packets, theformation of these batches may be internal to the PC-TCP modules,whether or not such batches are formed at the software applicationlevel. For example, the PC-TCP module at the transmitter may receive theoriginal data units that are used to form the original packets via asoftware interface from the source application. The packets aresegmented into blocks of N packets as described above, and the packetsqueued for transmission. In one embodiment, as long as the sourceapplication provides data units sufficiently quickly to keep the queuefrom emptying (or from emptying for a threshold amount of time), thePC-TCP module stays in the first mode (i.e., prior to sending forwarderror correction packets) sending repair packets as needed based onfeedback information from the receiver. When there is a lull in thesource application providing data units, then the PC-TCP module declaresthat a batch has been completed, and enters the forward error correctionphase described above. In some examples, the batch formed by the PC-TCPmodule may, in fact, correspond to a batch of data units generated bythe source application as a result of a lull in the source applicationproviding data units to the PC-TCP module while it computes data unitsfor a next batch, thereby inherently synchronizing the batch processingby the source application and the PC-TCP modules.

In one such embodiment, the PC-TCP module remains in the forward errorcorrection mode for the declared batch until that entire batch has beensuccessfully reconstructed at the receiver. In another embodiment, ifthe source application begins providing new data units before thereceiver has provided feedback that the previous batch has beensuccessfully reconstructed, the transmitter PC-TCP module begins sendingoriginal packets for the next batch at a lower priority than repair orforward error correction packets for the previous batch. Such anembodiment may reduce the time to the beginning of transmission of thenext batch, and therefore reduces the time to successful delivery of thenext batch.

In the embodiments in which the source application does not necessarilyprovide the data in explicit batches, the receiver PC-TCP moduleprovides the data units in order to the destination application withoutnecessarily identifying the block or batch boundaries introduced at thetransmitter PC-TCP module. That is, in at least some implementations,the transmitter and receiver PC-TCP modules provide a reliable channelfor the application data units without exposing the block and batchstructure to the applications.

As described above for certain embodiments, the transmitter PC-TCPmodule reacts to missing packet feedback from the receiver PC-TCP moduleto send repair packets. Therefore, it should be evident that themechanism by which the receiver sends such feedback may affect theoverall behavior of the protocol. For example, in one example, thereceiver PC-TCP module sends a negative acknowledgment as soon as itobserves a missing packet. Such an approach may provide the lowestlatency for reconstruction of the block. However, as introduced above,missing packets may be the result of out-of-order delivery. Therefore, aless aggressive generation of missing packet feedback, for example, bydelay in transmission of a negative acknowledgment, may reduce thetransmission of unnecessary repair packets with only a minimal increasein latency in reconstruction of that block. However, such delay insending negative acknowledgments may have an overall positive impact onthe time to successfully reconstruct the entire block because laterblocks are not delayed by unnecessary repair packets. Alternativeapproaches to generation of acknowledgments are described below.

In some embodiments, at least some of the determination of when to sendrepair packets is performed at the transmitter PC-TCP. For example, thereceiver PC-TCP module may not delay the transmission of missing packetfeedback, and it is the transmitter PC-TCP module that delays thetransmission of a repair packet based on its weighing of the possibilityof the missing packet feedback being based on out-of-order delivery asopposed to erasure.

3.5 Protocol Parameters

Communication between two PC-TCP endpoints operates according toparameters, some of which are maintained in common by the endpoints, andsome of which are local to the sending and/or the receiving endpoint.Some of these parameters relate primarily to forward error correctionaspects of the operation. For example, such parameters include thedegree of redundancy that is introduced through the coding process. Asdiscussed below, further parameters related to such coding relate to theselection of packets for use in the combinations. A simple example ofsuch selection is segmentation of the sequence of input data units into“frames” that are then independently encoded. In addition to the numberof such packets for combination (e.g., frame length), other parametersmay relate to overlapping and/or interleaving of such frames of dataunits and/or linear combinations of such data units.

Further parameters relate generally to transport layer characteristicsof the communication approach. For example, some parameters relate tocongestion avoidance, for example, representing a size of a window ofunacknowledged packets, transmission rate, or other characteristicsrelated to the timing or number of packets sent from the sender to thereceiver of the PC-TCP communication.

As discussed further below, communication parameters (e.g., codingparameters, transport parameters) may be set in various ways. Forexample, parameters may be initialized upon establishing a sessionbetween two PC-TCP endpoints. Strategies for setting those parametersmay be based on various sources of information, for example, accordingto knowledge of the communication path linking the sender and receiver(e.g., according to a classification of path type, such as 3G wirelessversus cable modem), or experienced communication characteristics inother sessions (e.g., concurrent or prior sessions involving the samesender, receiver, communication links, intermediate nodes, etc.).Communication parameters may be adapted during the course of acommunication session, for example, in response to observedcommunication characteristics (e.g., congestion, packet loss, round-triptime, etc.)

3.6 Transmission Control

Some aspects of the PC-TCP approaches relate to control of transmissionof packets from a sender to a receiver. These aspects are generallyseparate from aspects of the approach that determine what is sent in thepackets, for example, to accomplish forward error correction,retransmission, or the order in which the packets are sent (e.g.,relative priority of forward error correction packets versionretransmission packets). Given a queue of packets that are ready fortransmission from the sender to the receiver, these transmission aspectsgenerally relate to flow and/or congestion control.

3.6.1 Congestion Control

Current variants of TCP, including binary increase congestion control(BIC) and cubic-TCP, have been proposed to address the inefficiencies ofclassical TCP in networks with high losses, large bandwidths and longround-trip times. BIC-TCP and CUBIC algorithms have been used because oftheir stability. After a backoff, BIC increases the congestion windowlinearly then logarithmically to the window size just before backoff(denoted by W_(ax)) and subsequently increases the window in ananti-symmetric fashion exponentially then linearly. CUBIC increases thecongestion window following backoff according to a cubic function withinflection point at W_(max). These increase functions cause thecongestion window to grow slowly when it is close to W_(max), promotingstability. On the other hand, other variants such as HTCP and FAST TCPhave the advantage of being able to partially distinguish congestion andnon-congestion losses through the use of delay as a congestion signal.

An alternative congestion control approach is used in at least someembodiments. In some such embodiments, we identify a concave portion ofthe window increase function as W_(concave)(t)=W_(max)+c₁(t−k)³ and aconvex portion of the window increase function as W_(convex)(t)=W_(max)+c₂(t−k)³ where c₁ and c₂ are positive tunable parameters and

$k = \sqrt[3]{( {( {{W_{-}\max} - W} )\text{/}c_{1}} )}$

and W is the window size just after backoff.

This alternative congestion control approach can be flexibly tuned fordifferent scenarios. For example, a larger value of c₁ causes thecongestion window to increase more rapidly up to W_(max) and a largevalue of c₂ causes the congestion window to increase more rapidly beyondW_(max).

Optionally, delay is used as an indicator to exit slow start and move tothe more conservative congestion avoidance phase, e.g. when a smoothedestimate of RTT exceeds a configured threshold relative to the minimumobserved RTT for the connection. We can also optionally combine theincrease function of CUBIC or other TCP variants with the delay-basedbackoff function of HTCP.

In some embodiments, backoff is smoothed by allowing a lower rate oftransmission until the number of packets in flight decreases to the newwindow size. For instance, a threshold, n, is set such that once npackets have been acknowledged following a backoff, then one packet isallowed to be sent for every two acknowledged packets, which is roughlyhalf of the previous sending rate. This is akin to a hybrid window andrate control scheme.

3.6.2 Transmission Rate Control 3.6.2.1 Pacing Control by Sender

In at least some embodiments, pacing is used to regulate and/or spreadout packet transmissions, making the transmission rate less bursty.While pacing can help to reduce packet loss from buffer overflows,previous implementations of pacing algorithms have not shown clearadvantages when comparing paced TCP implementations to non-paced TCPimplementations. However, in embodiments where the data packets arecoded packets as described above, the combination of packet coding andpacing may have advantages. For example, since one coded packet may beused to recover multiple possible lost packets, we can use coding tomore efficiently recover from any spread out packet losses that mayresult from pacing. In embodiments, the combination of packet coding andpacing may have advantages compared to uncoded TCP with selectiveacknowledgments (SACK).

Classical TCP implements end-to-end congestion control based onacknowledgments. Variants of TCP designed for high-bandwidth connectionsincrease the congestion window (and consequently the sending rate)quickly to probe for available bandwidth but this can result in burstsof packet losses when it overshoots, if there is insufficient bufferingin the network.

A number of variants of TCP use acknowledgment feedback to determineround-trip time and/or estimate available bandwidth, and they differ inthe mechanisms with which this information is used to control thecongestion window and/or sending rate. Different variants have scenariosin which they work better or worse than others.

In one general approach used in one or more embodiments, a communicationprotocol may use smoothed statistics of intervals betweenacknowledgments of transmitted packets (e.g., a smoothed “ack interval”)to guide a transmission of packets, for example, by controllingintervals (e.g., an average interval or equivalently an averagetransmission rate) between packet transmissions. Broadly, this guidingof transmission intervals is referred to herein as “pacing”.

In some examples, the pacing approach is used in conjunction with awindow-based congestion control algorithm. Generally, the congestionwindow controls the number of unacknowledged packets that can be sent,in some examples using window control approaches that are the same orsimilar to those used in known variants of the Transmission ControlProtocol (TCP). In embodiments, the window control approach is based onthe novel congestion control algorithms described herein.

A general advantage of one or more aspects is to improve functioning ofa communication system, for instance, as measured by total throughput,or delay and/or variation in delay. These aspects address a technicalproblem of congestion, and with it packet loss, in a network by using“pacing” to reduce that congestion.

An advantage of this aspect is that the separate control of pacing canprevent packets in the congestion window from being transmitted toorapidly compared to the rate at which they are getting through to theother side. Without separate pacing control, at least some conventionalTCP approaches would permit bursts of overly rapid transmission ofpackets, which might result in packet loss at an intermediate node onthe communication path. These packet losses may be effectivelyinterpreted by the protocol as resulting from congestion, resulting inthe protocol reducing the window size. However, the window size may beappropriate, for example, for the available bandwidth and delay of thepath, and therefore reducing the window size may not be necessary. Onthe other hand, reducing the peak transmission rate can have the effectof avoiding packet loss, for example, by avoiding overflow ofintermediate buffers on the path.

Another advantage of at least some implementations is prevention oflarge bursts of packet losses under convex window increase functions forhigh-bandwidth scenarios, by providing an additional finer level ofcontrol over the transmission process.

At least some implementations of the approach can leverage theadvantages of existing high-bandwidth variants of TCP such as H-TCP andCUBIC, while preventing large bursts of packet losses under their convexwindow increase functions and providing a more precise level of control.For example, pacing control may be implemented to pace the rate ofproviding packets from the existing TCP procedure to the channel, withthe existing TCP procedure typically further or separately limiting thepresentation of packets to the communication channel based, forinstance, on its window-based congestion control procedure.

In practice, a particular example in which separating pacing from windowcontrol has been observed to significantly outperform conventional TCPon 4G LTE.

Referring to FIG. 22, in one example, a source application 1010 passesdata to a destination application 1090 over a communication channel1050. Communication from the source application 1010 passes to atransport layer 1020, which maintains a communication session with acorresponding transport layer 1080 linked to the destination application1090. In general, the transport layers may be implemented as softwarethat executes on the same computer as their corresponding applications,however, it should be recognized that, for instance, through the use ofproxy approaches, the applications and the transport layer elements thatare shown may be split over separate coupled computers. In embodiments,when a proxy is running on a separate machine or device from theapplication, the application may use the transport layer on its machineto communicate with the proxy layer.

In FIG. 22, the transport layer 1020 at the source application includesa window control and retransmission element 1030. In someimplementations, this element implements a conventional TransportControl Protocol (TCP) approach, for instance, implementing H-TCP orCUBIC approaches. In other implementations, this element implements thenovel congestion control algorithms described herein. The transportlayer 1080 at the destination may implement a corresponding element1060, which may provide acknowledgments of packets to the window controland retransmission element 1030 at the source. In general, element 1030may implement a window-based congestion control approach based onacknowledgments that are received at the destination, however, it shouldbe understood that no particular approach to window control isessential, and in some implementations, element 1030 can be substitutedwith another element that implements congestion control using approachesother than window control.

Functionally, one may consider two elements of the protocol as beingloss recovery and rate/congestion control. Loss recovery can beimplemented either using conventional retransmissions or using coding oras a combination of retransmission and coding. Rate/congestion controlmay aim to avoid overrunning the receiver and/or the available channelcapacity, and may be implemented using window control with or withoutpacing, or direct rate control.

The channel 1050 coupling the transport layers, in general, may includelower layer protocol software at the source and destination, and aseries of communication links coupling computers and other network nodeson a path from the source to the destination.

As compared to conventional approaches, as shown in FIG. 10, a ratecontrol element 1040 may be on the path between the window control andretransmission element 1030 and the channel 1050. This rate controlelement may monitor acknowledgments that are received from thedestination, and may pass them on to the window control andretransmission element 1030, generally without delay. The rate controlelement 1040 receives packets for transmission on the channel 1050 fromthe window control and retransmission element 1030, and either passesthem directly to the channel 1050, or buffers them to limit a rate oftransmission onto the channel. For example, the rate control element1040 may require a minimum interval between successive packets, or maycontrol an average rate over multiple packets.

In embodiments, the acks that are transmitted on a return channel, fromthe destination to the source, may also be paced, and may also utilizecoding to recover from erasures and bursty losses. In embodiments,packet coding and transmission control of the acks may be especiallyuseful if there is congestion on the return channel.

In one implementation, the rate control element 1040 may maintain anaverage (i.e., smoothed) inter-packet delivery interval, estimated basedon the acknowledgment intervals (accounting for the number of packetsacknowledged in each ack). In some implementations this averaging may becomputed as a decaying average of past sample inter-arrival times. Thiscan be refined by incorporating logic for discarding large sample valuesbased on the determination of whether they are likely to have resultedfrom a gap in the sending times or losses in the packet stream, and bysetting configurable upper and lower limits on the estimated intervalcommensurate with particular characteristics of different knownnetworks. The rate control element 1040 may then use this smoothedinter-acknowledgment time to set a minimum inter-transmission time, forexample, as a fraction of the inter-acknowledgment time. This fractioncan be increased with packet loss and with rate of increase of RTT(which may be indicators that the current sending rate may be too high),and decreased with rate of decrease of RTT under low loss, e.g. using acontrol algorithm such as proportional control whose parameters can beadjusted to tradeoff between stability and responsiveness to change.Upper and lower limits on this fraction can be made configurableparameters, say 0.2 and 0.95. Transmission packets are then limited tobe presented to the channel 1050 with inter-transmission times of atleast this set minimum. In other implementations, inter-transmissionintervals are controlled to maintain a smoothed average interval or ratebased on a smoothed inter-acknowledgment interval or rate.

In addition to the short timescale adjustments of the pacing intervalwith estimated delivery interval, packet loss rate and RTT describedabove, there can also be a longer timescale control loop that modulatesthe overall aggressiveness of the pacing algorithm based on a smoothedloss rate calculated over a longer timescale, with, a higher loss rateindicating that pacing may be too aggressive. The longer timescaleadjustment can be applied across short duration connections by havingthe client maintain state across successive connections and includeinitializing information in subsequent connection requests. This longertimescale control may be useful for improving adaptation to diversenetwork scenarios that change dynamically on different timescales.

Referring to FIG. 23, in some implementations, the communication channel1050 spans multiple nodes 1161, 1162 in one or an interconnection ofcommunication networks 1151, 1152. In FIG. 11, the source application1010 is illustrated as co-resident with the transport layer 1020 on asource computer 1110, and similarly, the transport layer 1080 isillustrated as co-resident on a destination computer 1190 with thedestination application 1090.

It should be recognized that although the description above focuses on asingle direction of communication, in general, a bidirectionalimplementation would include a corresponding path from the destinationapplication to the source application. In some implementations, bothdirections include corresponding rate control elements 1040, while inother applications, only one direction (e.g., from the source to thedestination application) may implement the rate control. For example,introduction of the rate control element 1040 at a server, or anotherdevice or network node on the path between the source application andthe transport layer 1080 at the destination, may not requiremodification of the software at the destination.

3.6.2.2 Pacing by Receiver

As described above, the sender can use acks to estimate therate/interval with which packets are reaching the receiver, the lossrate and the rate of change of RTT, and adjust the pacing intervalaccordingly. However, this estimated information may be noisy if acksare lost or delayed. On the other hand, such information can beestimated more accurately at the receiver with OWTT in place of RTT. Bybasing the pacing interval on the rate of change of OWTT rather than itsactual value, the need for synchronized clocks on sender and receivermay be obviated. The pacing interval can be fed back to the sender byincluding it as an additional field in the acks. The choice as towhether the pacing calculations are done at the sender or the receiver,or done every n packets rather than upon every packet reception, mayalso be affected by considerations of sender/receiver CPU/load.

3.7 Error Control

Classical TCP performs poorly on networks with packet losses. Congestioncontrol can be combined with coding such that coded packets are sentboth for forward error correction (FEC) to provide protection against ananticipated level of packet loss, as well as for recovering from actuallosses indicated by feedback from the receiver.

While the simple combination of packet coding and congestion control hasbeen suggested previously, the prior art does not adequately account fordifferences between congestion-related losses, bursty and/or randompacket losses. Since congestion-related loss may occur as relativelyinfrequent bursts, it may be inefficient to protect against this type ofloss using FEC.

In at least some embodiments, the rates at which loss events occur areestimated. A loss event may be defined as either an isolated packet lossor a burst of consecutive packet losses. In some examples, the sourcePC-TCP may send FEC packets at the estimated rate of loss events, ratherthan the estimated rate of packet loss. This embodiment is an efficientway to reduce non-useful FEC packets, since it may not bedisproportionately affected by congestion-related loss.

In an exemplary embodiment, the code rate and/or packet transmissionrate of FEC can be made tunable in order to trade-off between the usefulthroughput seen at the application layer (also referred to as goodput)and recovery delay. For instance, the ratio of the FEC rate to theestimated rate of loss events can be made a tunable parameter that isset with a priori knowledge of the underlying communications paths ordynamically adjusted by making certain measurements of the underlyingcommunications paths.

In another exemplary embodiment, the rate at which loss bursts of up toa certain length occur may be estimated, and appropriate burst errorcorrecting codes for FEC, or codes that correct combinations of burstand isolated errors, may be used.

In another exemplary embodiment, the FEC for different blocks can beinterleaved to be more effective against bursty loss.

In other exemplary embodiments, data packets can be sent preferentiallyover FEC packets. For instance, FEC packets can be sent at a configuredrate or estimated loss rate when there are no data packets to be sent,and either not sent or sent at a reduced rate when there are datapackets to be sent. In one implementation, FEC packets are placed in aseparate queue which is cleared when there are data packets to be sent.

In other exemplary embodiments, the code rate/amount of FEC in eachblock and/or the FEC packet transmission rate can be made a tunablefunction of the block number and/or the number of packets in flightrelative to the number of unacknowledged degrees of freedom of theblock, in addition to the estimated loss rate. FEC packets for laterblocks can be sent preferentially over FEC for earlier blocks, so as tominimize recovery delay at the end of a connection, e.g., the number ofFEC packets sent from each block can be a tunable function of the numberof blocks from the latest block that has not been fully acknowledged.The sending interval between FEC packets can be an increasing functionof the number of packets in flight relative to the number ofunacknowledged degrees of freedom of the corresponding block, so as totrade-off between sending delay and probability of losing FEC packets inscenarios where packet loss probability increases with transmissionrate.

In other exemplary embodiments, a variable randomly chosen fraction ofthe coding coefficients of a coded packet can be set to 1 or 0 in orderto reduce encoding complexity without substantially affecting erasurecorrection performance. In a systematic code, introducing 0 coefficientsonly after one or more densely coded packets (i.e. no or few 0coefficients) may be important for erasure correction performance. Forinstance, an initial FEC packet in a block could have each coefficientset to 1 with probability 0.5 and to a uniformly random value from thecoding field with probability 0.5. Subsequent FEC packets in the blockcould have each coefficient set to 0 with probability 0.5 and touniformly random value with probability 0.5.

3.7.1 Packet Reordering

As introduced above, packets may be received out of order on somenetworks, for example, due to packets traversing multiple paths,parallel processing in some networking equipment, reconfiguration of apath (e.g., handoff in cellular networks). Generally, conventional TCPreacts to out of order packets by backing off the size of the congestionwindow. Such a back-off may unnecessarily hurt performance if there isno congestion necessitating a back-off.

In some embodiments, in an approach to handling packet reordering thatdoes not result from congestions, a receiver observing a gap in thesequence numbers of its received packets may delay sending anacknowledgment for a limited time. When a packet is missing, thereceiver does not immediately know if the packet has been lost (erased),or merely reordered. The receiver delays sending an acknowledgment thatindicates the gap to see if the gap is filled by subsequent packetarrivals. In some examples, upon observing a gap, the receiver starts afirst timer for a configurable “reordering detection” time interval,e.g. 20 ms. If a packet from the gap is subsequently received withinthis time interval, the receiver starts a second timer for aconfigurable “gap filling” time interval, e.g. 30 ms. If the first timeror the second timer expire prior to the gap being filled, anacknowledgment that indicates the gap is sent to the source.

Upon receiving the acknowledgment that indicates the gap in receivedpackets the source, in at least some embodiments, the sender determineswhether a repair packet should be sent to compensate for the gap in thereceived packets, for example, if a sufficient number of FEC packetshave not already been sent.

In another aspect, a sender may store relevant congestion control stateinformation (including the congestion window) prior to back-off, and arecord of recent packet losses. If the sender receives an ack reportinga gap/loss and then subsequently one or more other acks reporting thatthe gap has been filled by out of order packet receptions, any back-offcaused by the earlier ack can be reverted by restoring the stored statefrom before back-off

In another aspect, a sender observing a gap in the sequence numbers ofits received acks may delay congestion window back-off for a limitedtime. When an ack is missing, the sender does not immediately know if apacket has been lost or if the ack is merely reordered. The senderdelays backing off its congestion window to see if the gap is filled bysubsequent ack arrivals. In some examples, upon observing a gap, thesender starts a first timer for a configurable “reordering detection”time interval, e.g. 20 ms. If an ack from the gap is subsequentlyreceived within this time interval, the sender starts a second timer fora configurable “gap filling” time interval, e.g. 30 ms. If the firsttimer or the second timer expire prior to the gap being filled,congestion window back-off occurs

In some examples, instead of using time intervals, packet sequencenumbers are used. For example, sending of an ack can be delayed until apacket which is a specified number of sequence numbers ahead of thereference lost packet is received. Similarly, backing off can be delayeduntil an acknowledgment of a packet which is a specified number ofsequence numbers ahead of the reference lost packet is received. In someexamples, these approaches have the advantage of being able to take intoaccount subsequently received/acknowledged reordered packets by shiftingthe sequence number of the reference lost packet as holes in the packetsequence get filled.

These methods for correcting packet reordering may be especially usefulfor multipath versions of the protocol, where there may be a largeamount of reordering.

3.7.2 Acknowledgments 3.7.2.1 Delayed Acknowledgments

In at least some implementations, conventional TCP sends oneacknowledgment for every two data packets received. Such delayed ackingreduces ack traffic compared to sending an acknowledgment for every datapacket. This reduction in ack traffic is particularly beneficial whenthere is contention on the return channel, such as in Wi-Fi networks,where both data and ack transmissions contend for the same channel.

It is possible to reduce ack traffic further by increasing the ackinterval to a value n>2, i.e. sending one acknowledgment for every ndata packets. However, reducing the frequency with which acks arereceived by the sender can cause delays in transmission (when thecongestion window is full) or back-off (if feedback on losses isdelayed), which can hurt performance.

In one aspect, the sender can determine whether, or to what extent,delayed acking should be allowed based in part on its remainingcongestion window (i.e. its congestion window minus the number ofunacknowledged packets in flight), and/or its remaining data to be sent.For example, delayed acking can be disallowed if there is any packetloss, or if the remaining congestion window is below some (possiblytunable) threshold. Alternatively, the ack interval can be reduced withthe remaining congestion window. As another example, delayed acking canbe allowed if the amount of remaining data to be sent is smaller thanthe remaining congestion window, but disallowed for the last remainingdata packet so that there is no delay in acknowledging the last datapacket. This information can be sent in the data packets as a flagindicating whether delayed acking is allowed, or for example, as aninteger indicating the allowed ack interval.

Using relevant state information at the sender to influence delayedacking may allow an increase in the ack interval beyond the conventionalvalue of 2, while mitigating the drawbacks described above that a largerack interval across the board might have.

To additionally limit the ack delay, each time an ack is sent, a delayedack timer can be set to expire with a configured delay, say 25 ms. Uponexpiration of the timer, any data packets received since the last ackmay be acknowledged, even if fewer packets than the ack interval n havearrived. If no packets have been received since the last ack, an ack maybe sent upon receipt of the next data packet.

3.8 Parameter control

3.8.1 Initialization

In some embodiments, to establish a session parameters for the PC-TCPmodules are set to a predefined set of default parameters. In otherembodiments, approaches that attempt to select better initial parametersare used. Approaches include use of parameter values from otherconcurrent or prior PC-TCP sessions, parameters determined fromcharacteristics of the communication channel, for example, selected fromstored parameters associated with different types of channels, orparameters determined by the source or destination application accordingto the nature of the data to be transported (e.g., batch versus stream).

3.8.2 Tunable Coding

Referring to FIG. 24, in an embodiment in which parameters are “tuned”(e.g., through feedback from a receiver or on other considerations) aserver application 2410 is in communication with a client application2490 via a communication channel 2450. In one example, the serverapplication 2410 may provide a data stream encoding multimedia content(e.g., a video) that is accepted by the client application 2490, forexample, for presentation to a user of the device on which the clientapplication is executing. The channel 2450 may represent what istypically a series of network links, for example including links of oneor more types, including:

-   -   a link traversing private links on a server local area network,    -   a link traversing the public Internet,    -   a link traversing a fixed (i.e., wireline) portion of a cellular        telephone network,    -   and a link traversing a wireless radio channel to the user's        device (e.g., a cellular telephone channel or satellite link or        wireless LAN).

The channel 2450 may be treated as carrying a series of data units,which may but do not necessarily correspond directly to InternetProtocol (IP) packets. For example, in some implementations multipledata units are concatenated into an IP packet, while in otherimplementations, each data unit uses a separate IP packet or only partof an IP packet. It should be understood that in yet otherimplementations, the Internet Protocol is not used—the techniquesdescribed below do not depend on the method of passing the data unitsover the channel 2450.

A transmitter 2420 couples the server application 2410 to the channel2450, and a receiver 2480 couples the channel 2450 to the clientapplication 2490. Generally, the transmitter 2420 accepts input dataunits from the server application 2480. In general, these data units arepassed over the channel 2450, as well as retained for a period of timein a buffer 2422. From time to time, an error control (EC) component2424 may compute a redundancy data unit from a subset of the retainedinput data units in the buffer 2422, and may pass that redundancy dataunit over the channel 2450. The receiver 2480 accepts data units fromthe channel 2450. In general, the channel 2450 may erase and reorder thedata units. Erasures may correspond to “dropped” data units that arenever received at the receiver, as well as corrupted data units that arereceived, but are known to have irrecoverable errors, and therefore aretreated for the most part as dropped units. The receiver may retain ahistory of received input data units and redundancy data units in abuffer 2482. An error control component 2484 at the receiver 2480 mayuse the received redundancy data units to reconstruct erased input dataunits that may be missing in the sequence received over the channel. Thereceiver 2480 may pass the received and reconstructed input data unitsto the client application. In general, the receiver may pass these inputdata units to the client application in the order they were received atthe transmitter.

In general, if the channel has no erasures or reordering, the receivercan provide the input data units to the client application with delayand delay variation that may result from traversal characteristics ofthe channel. When data units are erased in the channel 2450, thereceiver 2480 may make use of the redundancy units in its buffer 2482 toreconstruct the erased units. In order to do so, the receiver may haveto wait for the arrival of the redundancy units that may be useful forthe reconstruction. The way the transmitter computes and introduces theredundancy data units generally affects the delay that may be introducedto perform the reconstruction.

The way the transmitter computes and introduces the redundancy dataunits as part of its forward error correction function can also affectthe complexity of the reconstruction process at the receiver, and theutilization of the channel. Furthermore, regardless of the nature of theway the transmitter introduces the redundancy data units onto thechannel, statistically, there may be erased data units for which thereis insufficient information in the redundancy data units to reconstructthe erased unit. In such cases, the error control component 2484 mayrequest a retransmission of information from the error control component2424 of the transmitter 2420. In general, this retransmitted informationmay take the form of further redundancy information that depends on theerased unit. This retransmission process introduces a delay before theerased unit is available to the receiver. Therefore, the way thetransmitter introduces the redundancy information also affects thestatistics such as how often retransmission of information needs to berequested, and with it the delay in reconstructing the erased unit thatcannot be reconstructed using the normally introduced redundancyinformation.

In some embodiments, the error control component 2484 may provideinformation to the error control component 2424 to affect the way thetransmitter introduces the redundancy information. In general, thisinformation may be based on one or more of the rate of (or moregenerally the pattern of) erasures on units on the channel, rate of (ormore generally timing pattern of) and the state of the available unitsin the buffer 2482 and/or the state of unused data in the clientapplication 2490. For example, the client application may provide a“play-out time” (e.g., in milliseconds) of the data units that thereceiver has already provided to the client application such that if thereceiver were to not send any more units, the client application wouldbe “starved” for input units at that time. Note that in otherembodiments, rather than or in addition to receiving information fromthe receiver, the error control component 2424 at the transmitter mayget feedback from other places, for example, from instrumented nodes inthe network that pass back congestion information.

Referring to FIG. 25, a set of exemplary ways that the transmitterintroduces the redundancy data units into the stream of units passedover the channel makes use of alternating runs of input data units andredundancy data units. In FIG. 25, the data units that are “in flight”on the channel 2450 are illustrated passing from left to right in thefigure. The transmitter introduces the units onto the channel assequences of p input units alternating with sequences of q redundancyunits. Assuming that the data units are the same sizes, this correspondsto a rate R=p/(p+q) code. In an example with p=4 and q=2 and the codehas rate R=2/3.

In a number of embodiments, the redundancy units are computed as randomlinear combinations of past input units. Although the description belowfocuses on such approaches, it should be understood that the overallapproach is applicable to other computations of redundancy information,for example, using low density parity check (LDPC) codes and other errorcorrection codes. In the approach shown in FIG. 25, each run of qredundancy units is computed as a function of the previous D inputunits, wherein general but not necessarily D>p. In some cases, the mostrecent d data units transmitted are not used, and therefore theredundancy data units are computed from a window of D-d input dataunits. In FIG. 25, d=2, D=10, and D-d=8. Note that because D-d>p, thewindows of input data units used for computation of the successive runsof redundancy units overlap, such that any particular input data unitwill, in general, contribute to redundancy data units in more than oneof the runs of q units on the channel.

In FIG. 25, as well as in FIGS. 26-27 discussed below, buffered inputdata units (i.e., in buffer 2422 shown in FIG. 24) are shown on the leftwith time running from the bottom (past) to the top (future), with eachset of D-d units used to compute a run of q redundant units illustratedwith arrows. The sequence of transmitted units, consisting of runs ofinput data units alternating with runs of redundant units, is shown withtime running from right to left (i.e., later packets on the left). Dataunits that have been received and buffered at the receiver are shown onthe right (oldest on the bottom), redundant units computed from runs ofD-d input units indicated next to arrows representing the ranges ofinput data units used to compute those data units. Data units and rangesof input data units that have not yet been received are illustratedusing dashed lines.

FIGS. 26 and 27 show different selections of parameters. In FIG. 26, p=2and q=1 and the code has a rate R=2/3, which is the same rate at theselection of parameters in FIG. 25. Also as in the FIG. 52 selection,d=2, D=10, and D-d=8. Therefore, a difference between FIG. 25 and FIG.26 is not necessarily a degree of forward error protection (although theeffect of burst erasures may be somewhat different in the two cases).More importantly, the arrangement in FIG. 26 generally provides a lowerdelay from the time of an erased data unit to the arrival of redundancyinformation to reconstruct that unit, as compared to the arrangement inFIG. 25. On the other hand, the complexity of processing at the receivermay be greater in the arrangement of FIG. 26 as compared to thearrangement of FIG. 24, in part because redundancy units informationuses multiple different subsets of the input data units, which mayrequire more computation when reconstructing an erased data unit.Turning to FIG. 27, at another extreme, a selection of parameters useslonger blocks with a selection D=8 and q=4. Again, this code has a rateR=2/3. In general, this selection of parameters will incur greater delayin reconstruction of an erased data unit as compared to the selectionsof parameters shown in FIGS. 25 and 26. On the other hand,reconstruction of up to four erasures per block of D=8 input data unitsis relatively less complex than would be required by the selectionsshown in FIGS. 25 and 26.

For a particular rate of code (e.g., rate R=2/3), in an example,feedback received may result in changes of the parameters, for example,between (p,q)=(2,1) or (4,2) or (8,4) depending on of the amount of databuffered at the receiver, and therefore depending on the tolerance ofthe receiver to reconstruction delay.

Note that it is not required that q=p(1−R)/R is an integer, as it is inthe examples shown in FIGS. 25-27. In some embodiments, the length ofthe run of redundant units varies between q=┌p (1−R)/R┐ and q=└p(1−R)/R┘so that the average is ave(q)=p(1−R)/R.

In a variant of the approach described above, different input data unitshave different “priorities” or “importances” such that they areprotected to different degrees than other input data units. For example,in video coding, data units representing an independently coded videoframe may be more important than data units representing adifferentially encoded video frame. For example, if the priority levelsare indexed i=1, 2, . . . , then a proportion ρ_(i)≤1, whereΣ_(i)ρ_(i)=1, of the redundancy data units may be computed using dataunits with priority≤i. For example, for a rate R code, with blocks ofinput data units of length p, on average ρ₁ p(1−R)/R redundancy dataunits per block are computed from input data units with priority≤i.

The value of D should generally be no more than the target playout delayof the streaming application minus an appropriate margin forcommunication delay variability. The playout delay is the delay betweenthe time a message packet is transmitted and the time it should beavailable at the receiver to produce the streaming application output.It can be expressed in units of time, or in terms of the number ofpackets transmitted in that interval. D can be initially set based onthe typical or desired playout delay of the streaming application, andadapted with additional information from the receiver/application.Furthermore, choosing a smaller value reduces the memory and complexityat the expense of erasure correction capability.

The parameter d specifies the minimum separation between a messagepacket and a parity involving that message packet. Since a parityinvolving a message packet that has not yet been received is not usefulfor recovering earlier message packets involved in that parity, settinga minimum parity delay can improve decoding delay when packet reorderingis expected/observed to occur, depending partly also on the parityinterval.

Referring to FIG. 28, in an example implementation making use of theapproaches described above, the server application 2410 is hosted withthe transmitter 2420 at a server node 810, and the client application2490 is hosted at one or a number of client nodes 891 and 892. Althougha wide variety of types of data may be transported using the approachesdescribed above, one example is streaming of encoded multimedia (e.g.,video and audio) data. The communication channel 2450 (see FIG. 24) ismade up in this illustration as a path through one or more networks851-852 via nodes 861-862 in those respective networks. In someimplementations, the receiver is hosted at a client node 891 beinghosted on the same device as the client application 490.

3.8.3 Cross-Session Parameter Control

In some embodiments, the control of transport layer sessions usesinformation across connections, for example, across concurrent sessionsor across sessions occurring at different times.

Standard TCP implements end-to-end congestion control based onacknowledgments. A new TCP connection that has started up but not yetreceived any acknowledgments uses initial configurable values for thecongestion window and retransmission timeout. These values may be tunedfor different types of network settings.

Some applications, for instance web browser applications, may usemultiple connections between a client application (e.g., the browser)and a server application (e.g., a particular web server application at aparticular server computer). Conventionally, when accessing theinformation to render a single web “page”, the client application maymake many separate TCP sessions between the client and server computers,and using conventional TCP control, each session is controlledsubstantially independently. This independent control includes separatecongestion control.

One approach to addressing technical problems that are introduced byhaving such multiple sessions is the SPDY Protocol (see, e.g., SPDYProtocol—Draft 3.1, accessible athttp://www.chromium.org/spdy/spdy-protocol/spdy-protocol-draft3-1). TheSPDY protocol is an application layer protocol that manipulates HTTPtraffic, with particular goals of reducing web page load latency andimproving web security. Generally, SPDY effectively provides a tunnelfor the HTTP and HTTPS protocols. When sent over SPDY, HTTP requests areprocessed, tokenized, simplified and compressed. The resulting trafficis then sent over a single TCP session, thereby avoiding problems andinefficiencies involved in use of multiple concurrent TCP sessionsbetween a particular client and server computer.

In a general aspect, a communication system maintains informationrelated to communication between computers or network nodes. Forexample, the maintained information can include bandwidth to and/or fromthe other computer, current or past congestion window sizes, pacingintervals, packet loss rates, round-trip time, timing variability, etc.The information can include information for currently active sessionsand/or information about past sessions. One use of the maintainedinformation may be to initialize protocol parameters for a new sessionbetween computers for which information has been maintained. Forexample, the congestion window size or a pacing rate for a new TCP orUDP session may be initialized based on the congestion window size,pacing interval, round-trip time and loss rate of other concurrent orpast sessions.

Referring to FIG. 29, communication system 1200 maintains informationregarding communication sessions between endpoints. For example, thesecommunication sessions pass via a network 1250, and may pass between aserver 1210, or a proxy 1212 serving one or more servers 1214, and aclient 1290. In various embodiments, this information may be saved invarious locations. In some implementations, a client 1290 maintainsinformation about current or past connections. This information may bespecific to a particular server 1210 or proxy 1212. This information mayalso include aggregated information. For example, in the case of asmartphone on a cellular telephone network, some of the information maybe generic to connections from multiple servers and may representcharacteristics imposed by the cellular network rather than a particularpath to a server 1210. In some implementations, a server 1210 or proxy1212 may maintain the information based on its past communication withparticular clients 1290. In some examples, the clients and servers mayexchange the information such that is it distributed throughout thesystem 1200. In some implementations, the information may be maintainedin databases that are not themselves endpoints for the communicationsessions. For instance, it may be beneficial for a client withoutrelevant stored information to retrieve information from an externaldatabase.

In one use scenario, when a client 1290 seeks to establish acommunication session (e.g., a transport layer protocol session), itconsults its communication information 1295 to see if it has currentinformation that is relevant to the session it seeks to establish. Forexample, the client may have other concurrent sessions with a serverwith which it wants to communicate, or with which it may have recentlyhad such sessions. As another example, the client 1290 may useinformation about other concurrent or past sessions with other servers.When the client 1290 sends a request to a server 1210 or a proxy 1212 toestablish a session, relevant information for that session is also madeavailable to one or both of the endpoints establishing the session.There are various ways in which the information may be made available tothe server. For example, the information may be included with therequest itself. As another example, the server may request theinformation if it does not already hold the information in itscommunication information 1215. As another example, the server mayrequest the information from a remote or third party database, which hasbeen populated with information from the client or from servers thathave communicated with the client. In any case, the communicationsession between the client and the server is established usingparameters that are determined at least in part by the communicationinformation available at the client and/or server.

In some examples, the communication session may be established usinginitial values of packet pacing interval, congestion window,retransmission timeout and forward error correction. Initial valuessuitable for different types of networks (e.g. Wi-Fi, 4G), networkoperators and signal strength can be prespecified, and/or initial valuesfor successive connections can be derived from measured statistics ofearlier connections between the same endpoints in the same direction.For example:

-   -   The initial congestion window can be increased from its default        value if the packet throughput of the previous connection is        sufficiently larger than the ratio of the default initial        congestion window to the minimum round-trip time of the previous        connection. The congestion window can subsequently be adjusted        downwards if the initially received acks from the new connection        indicate that the available rate has decreased compared to the        previous connection.    -   The initial pacing interval can be set e.g. as MAX        (k1*congestion window/previous round-trip time, k2/previous        packet throughput), where k1 and k2 are configurable parameters,        or, with receiver pacing, as k* previous pacing interval, where        k increases with the loss rate of the previous connection.    -   Forward error correction parameters such as code rate can be set        as k*previous loss rate, where k is a configurable parameter.        The initial retransmission timeout can be increased from its        default value if the minimum round-trip time of the previous        connection is larger.

3.9 Multi-Path

FIG. 30 shows the use of multiple paths between the server and client todeliver the packet information. These multiple paths may be over similaror different network technologies with similar or different averagebandwidth, round trip delay, packet jitter rate, packet loss rate andcost. Examples of multiple paths include wired/fiber networks,geostationary, medium and low earth orbit satellites, WiFi, and cellularnetworks. In this example, the transmission control layer can utilize asingle session to distribute the N packets in the block beingtransmitted over the multiple paths according to a variety of metrics(average bandwidth of each path, round trip delay of each path, packetjitter rate, packet loss rate of each path, and cost). The N packets tobe transmitted in each block can be spread across each path in a mannerthat optimizes the overall end-to-end throughput and costs betweenserver and client. The number of packets sent on each path can bedynamically controlled such that the average relative proportions ofpackets sent on each path are in accordance with the average relativeavailable bandwidths of the paths, e.g. using back pressure-type controlwhereby packets are scheduled so as to approximately equalize queuelengths associated with the different paths.

For each path, the algorithms described above that embody transmissionand congestion control, forward error correction, sender based pacing,receiver based pacing, stream based parameter tuning, detection andcorrection for missing and out of order packets, use of informationacross multiple TCP connections, fast connection start and stop, TCP/UDPfallback, cascaded coding, recoding by intermediate nodes, and coding ofthe ACKs can be employed to improve the overall end-to-end throughputover the multiple paths between the source node and destination node.When losses are detected and FEC is used, the extra coded packets can besent over any or all of the paths. For instance, coded packets sent torepair losses can be sent preferentially over lower latency paths toreduce recovery delay. The destination node will decode any N of packetsthat are received over all of the paths and assemble them into a blockof N original packets by recreating any missing packets from the onesreceived. If less than N different coded packets are received across allpaths, then the destination node will request the number of missingpackets x where x=N−number of packets received be retransmitted. Any setof x different coded packet can be retransmitted over any path and thenused to reconstruct the missing packets in the block of N.

When there are networks with large differences in round trip time (RTT)latencies, the packets received over the lower RTT latencies will needto be buffered at the receiver in order to be combined with the higherRTT latency packets. The choice of packets sent on each path can becontrolled so as to reduce the extent of reordering and associatedbuffering on the receiver side, e.g. among the packets available to besent, earlier packets can be sent preferentially on higher latency pathsand later packets can be sent preferentially on lower latency paths.

Individual congestion control loops may be employed on each path toadapt to the available bandwidth and congestion on the path. Anadditional overall congestion control loop may be employed to controlthe total sending window or rate across all the paths of a multi-pathconnection, for fairness with single-path connections.

Referring to FIG. 31a , a communication system utilizes a first,satellite data path 3102 having a relatively high round trip timelatency and a second, DSL data path 3104 having a relatively low roundtrip time latency. When a user application 3106 sends a request tostream video content, a content server 3108 (e.g., video streamingservice) provides some or all of the requested video content to a remoteproxy 3110 which generates encoded video content 3112 for transmissionto the user application 3106. Based on the RTT latencies of the firstdata path 3102 and the second data path 3104, the remote proxy 3110splits the encoded video content 3112 into an initial portion 3114(e.g., the first 5 seconds of video content) and a subsequent portion3116 (e.g., the remaining video content). The remote proxy 3110 thencauses transmission of the initial portion 3114 over the second, lowlatency data path 3104 and transmission of the subsequent portion 3116over the first, high latency data path 3102.

Referring to FIG. 31B, due to the lower latency of the second data path3104, the initial portion 3114 of the video content arrives at the localproxy 3118 quickly, where it is decoded and sent to the user application3106 for presentation to a viewer. The subsequent portion 3116 of thevideo content is still traversing the first, high latency data path 3102at the time that presentation of the initial portion 3114 of the videocontent to the viewer commences.

Referring to FIG. 31C, during presentation of the decoded initialportion 3114 of video content to the viewer, the subsequent portion 3116of the video content arrives at the local proxy 3118 where it is decodedand sent to the user application 3106 before presentation of the initialportion 3114 of the video content to the viewer is complete. In someexamples, sending the initial portion 3114 of the video content over thelow latency data path 3104 and sending a subsequent portion 3116 of thevideo content over the high latency data path 3102 avoids lengthy waittimes between when a user requests a video and when the user sees thevideo (as would be the case if using satellite only communication) whileminimizing data usage over the low latency data path (which may be morecostly to use).

In some examples, other types of messages may be preferentially sentover the low latency data path. For example, acknowledgment messages,retransmission messages, and/or other time critical messages may betransmitted over the low latency data path while other data messages aretransmitted over the higher latency data path.

In some examples, additional data paths with different characteristics(e.g., latencies) can also be included in the communication system, withmessages being balanced over any of a number of data paths based oncharacteristics of the messages (e.g., message type) and characteristicsof the data paths.

In some examples, the initial portion of the messages and the subsequentportion of the messages are determined based on messages that arecurrently available at the transmitter. As additional messages becomeavailable to the transmitter, the initial portion and the subsequentportion of the messages can dynamically change based on the additionalmessages being divided between the two portions. For example, at a firsttransmission opportunity, packets 1-10 are available for transmission.The system decides to transmit packet 1 over a first data path andpacket 10 over a second data path. By a second transmission opportunity,packets 11-12 have become available for transmission. Packet 2 is senton the first data path and packet 12 on the second.

Some examples of high latency data paths include medium earth orbitsatellite data paths and geosynchronous orbit satellite data paths. Someexamples of low latency data paths include digital subscriber line (DSL)data paths, cable internet based data paths, fiber optic data paths, andcellular data paths.

4 Alternatives and Implementations

In the document above, certain features of the packet coding andtransmission control protocols are described individually, or inisolation, but it should be understood that there are certain advantagesthat may be gained by combining multiple features together. Preferredembodiments for the packet coding and transmission control protocolsdescribed may depend on whether the transmission links and network nodestraversed between communication session end-points belong to certainfiber or cellular carriers (e.g. AT&T, T-Mobile, Sprint, Verizon, Level3) and/or end-user Internet Service Providers (ISPs) (e.g. AT&T,Verizon, Comcast, Time Warner, Century Link, Charter, Cox) or are overcertain wired (e.g. DSL, cable, fiber-to-the-curb/home (FTTx)) orwireless (e.g. WiFi, cellular, satellite) links. In embodiments, probetransmissions may be used to characterize the types of network nodes andtransmission links communication signals are traversing and the packetcoding and transmission control protocol may be adjusted to achievecertain performance. In some embodiments, data transmissions may bemonitored to characterize the types of network nodes and transmissionlinks communication signals are traversing and the packet coding andtransmission control protocol may be adjusted to achieve certainperformance. In at least some embodiments, quantities such asround-trip-time (RTT), one-way transmission times (OWTT), congestionwindow, pacing rate, packet loss rate, number of overhead packets, andthe like may be monitored continuously, intermittently, in response to atrigger signal or event, and the like. In at least some embodiments,combinations of probe transmissions and data transmissions may be usedto characterize network and communication session performance in realtime.

In at least some embodiments, network and communication parameters maybe stored in the end-devices of communication sessions and/or they maybe stored in network resources such as servers, switches, nodes,computers, databases and the like. These network and communicationparameters may be used by the packet coding and transmission controlprotocol to determine initial parameter settings for the protocol toreduce the time it may take to adjust protocol parameters to achieveadequate performance. In embodiments, the network and communicationparameters may be tagged and/or associated with certain geographicallocations, network nodes, network paths, equipment types, carriernetworks, service providers, types of transmission paths and the like.In embodiments, the end-devices may be configured to automaticallyrecord and/or report protocol parameter settings and to associate thosesettings with certain locations determined using GPS-type locationidentification capabilities resident in those devices. In embodiments,the end-devices may be configured to automatically record and/or reportprotocol parameters settings and to associate those settings withcertain carrier networks, ISP equipment traversed, types of wired and/orwireless links and the like.

In at least some embodiments, a packet coding and transmission controlprotocol as described above may adjust more than one parameter toachieve adequate or improved network performance. Improved networkperformance may be characterized by less delay in delivering datapackets, less delay in completing file transfers, higher quality audioand video signal delivery, more efficient use of network resources, lesspower consumed by the end-users, more end-users supported by existinghardware resources and the like.

In at least some embodiments, certain modules or features of the packetcoding and transmission control protocol may be turned on or offdepending on the data's path through a network. In some embodiments, theorder in which certain features are implemented or controlled may beadjusted depending on the data's path through a network. In someembodiments, the probe transmissions and/or data transmissions may beused in open-loop or closed-loop control algorithms to adjust theadjustable parameters and/or the sequence of feature implementation inthe packet coding and transmission control protocol.

It should be understood that examples that involve monitoring to controlthe protocol can, in general, involve aspects that are implemented atthe source, the destination, or at a combination of the source and thedestination. Therefore, it should be evident that although embodimentsare described above in which features are described as being implementedat particular endpoints, alternative embodiments involve implementationof those features at different endpoints. Also, as described above,monitoring to control the protocol can, in general, involve aspects thatare implemented intermediate nodes or points in the network. Therefore,it should be evident that although embodiments are described above inwhich features are described as being implemented at particularendpoints, alternative embodiments involve implementation of thosefeatures at different nodes, including intermediate nodes, throughoutthe network.

In addition to the use of monitored parameters for control of theprotocols, the data may be used for other purposes. For example, thedata may support network analytics that are used, for example, tocontrol or provision the network as a whole.

The PC-TCP approaches may be adapted to enhance existing protocols andprocedures, and in particular protocols and procedures used in contentdelivery, for example, as used in coordinated content delivery networks.For instance, monitored parameters may be used to direct a client to theserver or servers that can deliver an entire unit of content as soon aspossible rather than merely direct the client to a least loaded serveror to server accessible over a least congested path. A difference insuch a new approach is that getting an entire file as fast as possiblemay require packets to be sent from multiple servers and/or servers thatare not geographically the closest, over multiple links, and using newacknowledgment protocols that coordinate the incoming data whilerequiring a minimum of retransmissions or FEC overhead. Coordinating mayinclude waiting for gaps in strings of packets (out-of-order packets) tobe filled in by later arriving packets and/or by coded packets. Inaddition, the PC-TCP approaches may improve the performance of wireless,cellular, and satellite links, significantly improving the end-to-endnetwork performance.

Some current systems use “adaptive bit rates” to try to preserve videotransmission through dynamic and/or poorly performing links. In someinstances, the PC-TCP approaches described above replace adaptive bitrate schemes and may be able to present a very high data rate to a userfor a long period of time. In other instances, the PC-TCP approaches areused in conjunction with currently-available adaptive bit rate schemesto support higher data rates on average than could be supported byadaptive bit rate schemes alone. In some instances, the PC-TCPapproaches may include integrated bit rate adjustments as part of itsfeature set and may use any and/or all of the previously identifiedadjustable parameters and/or monitored parameters to improve theperformance of a combined PC-TCP and bit-rate adaptive solution.

Implementations of approaches described above may include softwareimplementations, which use software instructions stored onnon-transitory machine-readable media. The procedures and protocols asdescribed above in the text and figures are sufficient for one skilledin the art to implement them in such software implementations. In someexamples, the software may execute on a client node (e.g., a smartphone)using a general-purpose processor that implements a variety of functionson the client node. Software that executes on end nodes or intermediatenetwork nodes may use processors that are dedicated to processingnetwork traffic, for example, being embedded in network processingdevices. In some implementations, certain functions may be implementedin hardware, for example, using Application-Specific Integrated Circuits(ASICs), and/or Field Programmable Gate Arrays (FPGAs), thereby reducingthe load on a general purpose processor.

Note that in some diagrams and figures in this disclosure, networks suchas the internet, carrier networks, internet service provider networks,local area networks (LANs), metro area networks (MANs), wide areanetworks (WANs), storage area networks (SANs), backhaul networks,cellular networks, satellite networks and the like, may be depicted asclouds. Also note, that certain processes may be referred to as takingplace in the cloud and devices may be described as accessing the cloud.In these types of descriptions, the cloud should be understood to besome type of network comprising networking equipment and wireless and/orwired links.

The description above may refer to a client device communicating with aserver, but it should be understood that the technology and techniquesdescribed herein are not limited to those exemplary devices as theend-points of communication connections or sessions. The end-points mayalso be referred to as, or may be, senders, transmitters, transceivers,receivers, servers, video servers, content servers, proxy servers, cloudstorage units, caches, routers, switches, buffers, mobile devices,tablets, smart phones, handsets, computers, set-top boxes, modems,gaming systems, nodes, satellites, base stations, gateways, satelliteground stations, wireless access points, and the like. The devices atany of the end-points or intermediate nodes of communication connectionsor sessions may be commercial media streaming boxes such as thoseimplementing Apple TV, Roku, Chromecast, Amazon Fire, Slingbox, and thelike, or they may be custom media streaming boxes. The devices at theany of the end-points or intermediate nodes of communication connectionsor sessions may be smart televisions and/or displays, smart appliancessuch as hubs, refrigerators, security systems, power panels and thelike, smart vehicles such as cars, boats, busses, trains, planes, carts,and the like, and may be any device on the Internet of Things (IoT). Thedevices at any of the end-points or intermediate nodes of communicationconnections or sessions may be single-board computers and/or purposebuilt computing engines comprising processors such as ARM processors,video processors, system-on-a-chip (SoC), and/or memory such as randomaccess memory (RAM), read only memory (ROM), or any kind of electronicmemory components.

Communication connections or sessions may exist between two routers, twoclients, two network nodes, two servers, two mobile devices, and thelike, or any combination of potential nodes and/or end-point devices. Inmany cases, communication sessions are bi-directional so that bothend-point devices may have the ability to send and receive data. Whilethese variations may not be stated explicitly in every description andexemplary embodiment in this disclosure, it should be understood thatthe technology and techniques we describe herein are intended to beapplied to all types of known end-devices, network nodes and equipmentand transmission links, as well as to future end-devices, network nodesand equipment and transmission links with similar or improvedperformance.

In embodiments, methods and systems are provided for data collection inor relating to one or more machines deployed in an industrialenvironment using self-organized network coding for network transmissionof sensor data in a network. In embodiments, network coding may be usedto specify and manage the manner in which packets (including streams ofpackets as noted in various embodiments disclosed throughout thisdisclosure and the documents incorporated by reference) are relayed froma sender (e.g., a data collector, instrumentation system, computer, orthe like in an industrial environment where data is collected, such asfrom sensors or instruments on, in or proximal to industrial machines orfrom data storage in the environment) to a receiver (e.g., another datacollector (such as in a swarm or coordinated group), instrumentationsystem, computer, storage, or the like in the industrial environment, orto a remote computer, server, cloud platform, database, data pool, datamarketplace, mobile device (e.g., mobile phone, personal computer,tablet, or the like), or other network-connected device of system), suchas via one or more network infrastructure elements (referred to in somecases herein as nodes), such as access points, switches, routers,servers, gateways, bridges, connectors, physical interfaces and thelike, using one or more network protocols, such as IP-based protocols,TCP/IP, UDP, HTTP, Bluetooth, Bluetooth Low Energy, cellular protocols,LTE, 2G, 3G, 4G, 5G, CDMA, TDSM, packet-based protocols, streamingprotocols, file transfer protocols, broadcast protocols, multi-castprotocols, unicast protocols, and others. For situations involvingbi-directional communication, any of the above-referenced devices orsystems, or others mentioned throughout this disclosure, may play therole of sender or receiver, or both. Network coding may account foravailability of networks, including the availability of multiplealternative networks, such that a transmission may be delivered acrossdifferent networks, either separated into different components orsending the same components redundantly. Network coding may account forbandwidth and spectrum availability; for example, a given spectrum maybe divided (such as with sub-dividing spectrum by frequency, bytime-division multiplexing, and other techniques). Networks orcomponents thereof may be virtualized, such as for purposes ofprovisioning of network resources, specification of network coding for avirtualized network, or the like. Network coding may include a widevariety of approaches as described herein and shown in the figures.

In embodiments, one or more network coding systems or methods of thepresent disclosure may use self-organization, such as to configurenetwork coding parameters for one or more transmissions over one or morenetworks using an expert system, which may comprise a model-based system(such as automatically selecting network coding parameters orconfiguration based on one or more defined or measured parametersrelating to a transmission, such as parameters of the data or content tobe transmitted, the sender, the receiver, the available networkinfrastructure components, the conditions of the network infrastructure,the conditions of the industrial environment, or the like). A model may,for example, account for parameters relating to file size, numbers ofpackets, size of a stream, criticality of a data packet or stream, valueof a packet or stream, cost of transmission, reliability of atransmission, quality of service, quality of transmission, quality ofuser experience, financial yield, availability of spectrum, input/outputspeed, storage availability, storage reliability, and many others asnoted throughout this disclosure. In embodiments, the expert system maycomprise a rule-based system, where one or more rules is executed basedon detection of a condition or parameter, calculation of a variable, orthe like, such as based on any of the above-noted parameters. Inembodiments, the expert system may comprise a machine learning system,such as a deep learning system, such as based on a neural network, aself-organizing map, or other artificial intelligence approach(including any noted throughout this disclosure or the documentsincorporated by reference). A machine learning system in any of theembodiments of this disclosure may configure one or more inputs,weights, connections, functions (including functions of individualneurons or groups of neurons in a neural net) or other parameters of anartificial intelligence system. Such configuration may occur withiteration and feedback, optionally involving human supervision, such asby feeding back various metrics of success or failure. In the case ofnetwork coding, configuration may involve setting one or more codingparameters for a network coding specification or plan, such asparameters for selection of a network, selection one or more nodes,selection of data path, configuration of timers or timing parameters,configuration of redundancy parameters, configuration of coding types(including use of regenerating codes, such as for use of network codingfor distributed storage, such as in peer-to-peer networks, such as apeer-to-peer network of data collectors, or a storage network for adistributed ledger, as noted elsewhere in this disclosure), coefficientsfor coding (including linear algebraic coefficients), parameters forrandom or near-random linear network coding (including generation ofnear random coefficients for coding), session configuration parameters,or other parameters noted in the network coding embodiments describedbelow, throughout this disclosure, and in the documents incorporatedherein by reference. For example, a machine learning system mayconfigure the selection of a protocol for a transmission, the selectionof what network(s) will be used, the selection of one or more senders,the selection of one or more routes, the configuration of one or morenetwork infrastructure nodes, the selection of a destination receiver,the configuration of a receiver, and the like. In embodiments, each oneof these may be configured by an individual machine learning system, orthe same system may configure an overall configuration by adjustingvarious parameters of one or more of the above under iteration, througha series of trials, optionally seeded by a training set, which may bebased on human configuration of parameters, or by model-based and/orrule-based configuration. Feedback to a machine learning system maycomprise various measures, including transmission success or failure,reliability, efficiency (including cost-based, energy-based and othermeasures of efficiency, such as measuring energy per bit transmitted,energy per bit stored, or the like), quality of transmission, quality ofservice, financial yield, operational effectiveness, success atprediction, success at classification, and others. In embodiments, amachine learning system may configure network coding parameters bypredicting network behavior or characteristics and may learn to improveprediction using any of the techniques noted above. In embodiments, amachine learning system may configure network coding parameters byclassification of one or more network elements and/or one or morenetwork behaviors and may learn to improve classification, such as bytraining and iteration over time. Such machine-based prediction and/orclassification may be used for self-organization, including bymodel-based, rule-based, and machine learning-based configuration. Thus,self-organization of network coding may use or comprise variouscombinations or permutations of model-based systems, rule-based systems,and a variety of different machine-learning systems (includingclassification systems, prediction systems, and deep learning systems,among others).

As described in US patent application 2017/0013065, entitled“Cross-session network communication configuration,” network coding mayinvolve methods and systems for data communication over a data channelon a data path between a first node and a second node and may includemaintaining data characterizing one or more current or previous datacommunication connections traversing the data channel and initiating anew data communication connection between the first node and the secondnode including configuring the new data communication connection atleast in part according to the maintained data. The maintained data maycharacterize one or more data channels on one or more data paths betweenthe first node and the second node over which said one or more currentor previous data communication connections pass. The maintained data maycharacterize an error rate of the one or more data channels. Themaintained data may characterize a bandwidth of the one or more datachannels. The maintained data may characterize a round trip time of theone or more data channels. The maintained data may characterizecommunication protocol parameters of the one or more current or previousdata communication connections.

The communication protocol parameters may include one or more of acongestion window size, a block size, an interleaving factor, a portnumber, a pacing interval, a round trip time, and a timing variability.The communication protocol parameters may include two or more of acongestion window size, a block size, an interleaving factor, a portnumber, a pacing interval, a round trip time, and a timing variability.

The maintained data may characterize forward error correction parametersassociated with the one or more current or previous data communicationconnections. The forward error correction parameters may include a coderate. Initiating the new data communication connection may includeconfiguring the new data communication connection according to firstdata of the maintained data, the first data is maintained at the firstnode, and initiating the new data communication connection includesproviding the first data from the first node to the second node forconfiguring the new data communication connection.

Initiating the new data communication connection may include configuringthe new data communication connection according to first data of themaintained data, the first data is maintained at the first node, andinitiating the new data communication connection includes accessingfirst data at the first node for configuring the new data communicationconnection. Any one of these elements of maintained data, includingvarious parameters of communication protocol, error correctionparameters, connection parameters, and others, may be provided to theexpert system for supporting self-organization of network coding,including for execution of rules to set network coding parameters basedon the maintained data, for population of a model, or for configurationof parameters of a neural net or other artificial intelligence system.

Initiating the new data communication connection may include configuringthe new data communication connection according to first data of themaintained data, the first data being maintained at the first node, andinitiating the new data communication connection includes accepting arequest from the first node for establishing the new data communicationconnection between the first node and the second node, includingreceiving, at the second node, at least one message from the first nodecomprising the first data for configuring said connection. The methodmay include maintaining the new data communication connection betweenthe first node and the second node, including maintaining communicationparameters, including initializing said communication parametersaccording the first data received in the at least one message from thefirst node.

Maintaining the new data communication connection may include adaptingthe communication parameters according to feedback from the first node.The feedback from the first node may include feedback messages receivedfrom the first node. The feedback may include feedback derived from aplurality of feedback messages received from the first node. Feedbackmay relate to any of the types of feedback noted above, and may be usedfor self-organizing the data communication connection using the expertsystem.

In some examples, one or more training communication connections over adata channel on a data path are employed prior to establishment of datacommunication connections over the data channel on the data path. Thetraining communication connections are used to collect information aboutthe data channel which is then used when establishing the datacommunication connections. In other examples, no training communicationconnections are employed and information about the data channel isobtained from one or more previous or current data communicationconnection over the data channel on the data path.

In embodiments, a method for data communication over a data channel on adata path between a first node and a second node includes maintainingdata characterizing one or more current or previous data communicationconnections traversing the data channel; and initiating a new datacommunication connection between the first node and the second nodeincluding configuring the new data communication connection at least inpart according to the maintained data, wherein the configuration of thenew data communication connection is configured by an expert system.

In embodiments, the expert system uses at least one of a rule and amodel to set a parameter of the configuration.

In embodiments, the expert system is a machine learning system thatiteratively configures at least one of a set of inputs, a set ofweights, and a set of functions based on feedback relating to the datachannel.

In embodiments, the expert system takes a plurality of inputs from adata collector that accepts data about a machine operating in anindustrial environment.

As described in US patent application 2017/0012861, entitled “Multi-pathnetwork communication,” self-organized network coding under control ofan expert system may involve methods and systems for data communicationbetween a first node and a second node over a number of data pathscoupling the first node and the second node and may include transmittingmessages between the first node and the second node over the number ofdata paths, including transmitting a first subset of the messages over afirst data path of the number of data paths and transmitting a secondsubset of the messages over a second data path of the number of datapaths. In situations where the first data path has a first latency andthe second data path has a second latency substantially larger than thefirst latency, and messages of the first subset of the messages arechosen to have first message characteristics and messages of the secondsubset are chosen to have second message characteristics, different fromthe first message characteristics.

Messages having the first message characteristics, targeted for datapaths of lower latency, may include time critical messages; for example,in an industrial environment, messages relating to a critical faultcondition of a machine (e.g., overheating, excessive vibration, or anyof the other fault conditions described throughout this disclosure) orrelating to a safety hazard, or a time-critical operational step onwhich other processes depend (e.g., completion of a catalytic reaction,completion of a sub-assembly, or the like in a high-value, high-speedmanufacturing process, a refining process, or the like) may bedesignated as time critical (such as by a rule that can be parsed orprocessed by a rules engine) or may be learned to be time-critical bythe expert system, such as based on feedback regarding outcomes overtime, including outcomes for similar machines having similar data insimilar industrial environments. The first subset of the messages andthe second subset of the messages may be determined from a portion ofthe messages available at the first node at a time of transmission. At asubsequent time of transmission, additional messages made available tothe first node may be divided into the first subset and the secondsubset based on message characteristics associated with the additionalmessages. Division into subsets and selection of what subsets aretargeted to what data path may be undertaken by an expert system.Messages having the first message characteristics may be associated withan initial subset of a data set and messages having the second messagecharacteristics may be associated with a subsequent subset of the dataset. The methods and systems described herein for selecting inputs fordata collection and for multiplexing data may be organized, such as byan expert system, to configure inputs for the alternative channels, suchas by providing streaming elements that have real-time significance tothe first data path and providing other elements, such as for long-term,predictive maintenance, to the other data path. In embodiments, themessages of the second subset may include messages that are at most nmessages ahead of a last acknowledged message in a sequentialtransmission order associated with the messages, wherein n is determinedbased on a buffer size at one of the first and second nodes.

Messages having the first message characteristics may includeacknowledgment messages and messages having the second messagecharacteristics may include data messages. Messages having the firstmessage characteristics may include supplemental data messages. Thesupplemental data messages may include redundancy data and messageshaving the second message characteristics may include original datamessages. The first data path may include a terrestrial data path andthe second data path may include a satellite data path. The terrestrialdata path may include one or more of a cellular data path, a digitalsubscriber line (DSL) data path, a fiber optic data path, a cableinternet based data path, and a wireless local area network data path.The satellite data path may include one or more of a low earth orbitsatellite data path, a medium earth orbit satellite data path, and ageostationary earth orbit satellite data path. The first data path mayinclude a medium earth orbit satellite data path or a low earth orbitsatellite data path and the second data path may include a geostationaryorbit satellite data path.

The method may further include, for each path of the number of datapaths, maintaining an indication of successful and unsuccessful deliveryof the messages over the data path and adjusting a congestion window forthe data path based on the indication, which may occur under control ofan expert system, including based on feedback of outcomes of a set oftransmissions. The method may further include, for each path of thenumber of data paths, maintaining, at the first node, an indication ofwhether a number of messages received at the second node is sufficientto decode data associated with the messages, wherein the indication isbased on feedback received at the first node over the number of datapaths.

In another general aspect, a system for data communication between anumber of nodes over a number of data paths coupling the number of nodesincludes a first node configured to transmit messages to a second nodeover the number of data paths including transmitting a first subset ofthe messages over a first data path of the number of data paths, andtransmitting a second subset of the messages over a second data path ofthe number of data paths.

In embodiments, the first subset of the messages and the second subsetof the messages for the respective data paths may be determined from aportion of the messages available at a first node at a time oftransmission. At a subsequent time of transmission, additional messagesmade available to the first node may be divided into a first subset anda second subset based on message characteristics associated with theadditional messages. Messages having the first message characteristicsmay be associated with an initial subset of a data set and messageshaving the second message characteristics may be associated with asubsequent subset of the data set.

In embodiments, the messages of the second subset may include messagesthat are at most n messages ahead of a last acknowledged message in asequential transmission order associated with the messages, wherein n isdetermined based on a receive buffer size at the second node. Messageshaving the first message characteristics may include acknowledgmentmessages and messages having the second message characteristics mayinclude data messages. Messages having the first message characteristicsmay include supplemental data messages. The supplemental data messagesmay include data messages including redundancy data and messages havingthe second message characteristics may include original data messages.

The first node may be further configured to, for each path of the numberof data paths, maintain an indication of successful and unsuccessfuldelivery of the messages over the data path and adjust a congestionwindow for the data path based on the indication. The first node may befurther configured to maintain an aggregate indication of whether anumber of messages received at the second node over the number of datapaths is sufficient to decode data associated with the messages and totransmit supplemental messages based on the aggregate indication,wherein the aggregate indication is based on feedback from the secondnode received at the first node over the number of data paths.

In embodiments, a method for data communication between a first node anda second node over a plurality of data paths coupling the first node andthe second node includes transmitting messages between the first nodeand the second node over the plurality of data paths includingtransmitting a first subset of the messages over a first data path ofthe plurality of data paths, and transmitting a second subset of themessages over a second data path of the plurality of data paths; whereinthe first data path has a first latency and the second data path has asecond latency substantially larger than the first latency, and messagesof the first subset of the messages are chosen to have first messagecharacteristics and messages of the second subset are chosen to havesecond message characteristics, different from the first messagecharacteristics, wherein the selection of the first and second subset ofmessage characteristics is performed automatically under control of anexpert system.

In embodiments, the expert system uses at least one of a rule and amodel to set a parameter of the selection.

In embodiments, the expert system is a machine learning system thatiteratively configures at least one of a set of inputs, a set ofweights, and a set of functions based on feedback relating to at leastone of the data paths. In embodiments, the expert system takes aplurality of inputs from a data collector that accepts data about amachine operating in an industrial environment.

As described in US patent application 2017/0012868, entitled “Multipleprotocol network communication,” self-organized network coding undercontrol of an expert system may involve methods and systems for datacommunication between a first node and a second node over one or moredata paths coupling the first node and the second node and may includetransmitting messages between the first node and the second node overthe data paths, including transmitting at least some of the messagesover a first data path using a first communication protocol,transmitting at least some of the messages over a second data path usinga second communication protocol, determining that the first data path isaltering a flow of messages over the first data path due to the messagesbeing transmitted using the first communication protocol, and inresponse to the determining, adjusting a number of messages sent overthe data paths, including decreasing a number of the messagestransmitted over the first data path and increasing a number of messagestransmitted over the second data path. Determination that the first datapath is altering a flow of messages and/or adjusting the number ofmessages sent over the data paths may occur under control of an expertsystem, such as a rule-based system, a model-based system, a machinelearning system (including deep learning) or a hybrid of any of those,where the expert system takes inputs relating to one or more of the datapaths, the nodes, the communication protocols used, or the like. Thedata paths may be among devices and systems in an industrialenvironment, such as instrumentation systems of industrial machines, oneor more mobile data collectors (optionally coordinated in a swarm), datastorage systems (including network-attached storage), servers and otherinformation technology elements, any of which may have or be associatedwith one or more network nodes. The data paths may be among any suchdevices and systems and devices and systems in a network of any kind(such as switches, routers, and the like) or between those and oneslocated in a remote environment, such as in an enterprise's informationtechnology system, in a cloud platform, or the like.

Determining that the first data path is altering the flow of messagesover the first data path may include determining that the first datapath is limiting a rate of messages transmitted using the firstcommunication protocol. Determining that the first data path is alteringthe flow of messages over the first data path may include determiningthat the first data path is dropping messages transmitted using thefirst communication protocol at a higher rate than a rate at which thesecond data path is dropping messages transmitted using the secondcommunication protocol. The first communication protocol may be the UserDatagram Protocol (UDP), and the second communication protocol may bethe Transmission Control Protocol (TCP), or vice versa. Other protocolsas described throughout this disclosure may be used.

The messages may be initially equally divided or divided according tosome predetermined allocation (such as by type, as noted in connectionwith other embodiments) across the first data path and the second datapath, such as using a load balancing technique. The messages may beinitially divided across the first data path and the second data pathaccording to a division of the messages across the first data path andthe second data path in one or more prior data communicationconnections. The messages may be initially divided across the first datapath and the second data path based on a probability that the first datapath will alter a flow of messages over the first data path due to themessages being transmitted using the first communication protocol.

The messages may be divided across the first data path and the seconddata path based on message type. The message type may include one ormore of acknowledgment messages, forward error correction messages,retransmission messages, and original data messages. Decreasing a numberof the messages transmitted over the first data path and increasing anumber of messages transmitted over the second data path may includesending all of the messages over the second path and sending none of themessages over the first path.

At least some of the number of data paths may share a common physicaldata path. The first data path and the second data path may share acommon physical data path. The adjusting of the number of messages sentover the number of data paths may occur during an initial phase of thetransmission of the messages. The adjusting of the number of messagessent over the number of data paths may repeatedly occur over a durationof the transmission of the messages. The adjusting of the number ofmessages sent over the number of data paths may include increasing anumber of the messages transmitted over the first data path anddecreasing a number of messages transmitted over the second data path.

In some examples, the parallel transmission over TCP and UDP is handleddifferently from conventional load balancing techniques, because TCP andUDP both share a low-level data path and nevertheless have verydifferent protocol characteristics.

In some examples, approaches respond to instantaneous network behaviorand learn the network's data handling policy and state by probing forchanges. In an industrial environment, this may include learningpolicies relating to authorization to use aspects of a network; forexample, a SCADA system may allow a data path to be used only by alimited set of authorized users, services, or applications, because ofthe sensitivity of underlying machines or processes that are undercontrol (including remote control) via the SCADA system and concern overpotential for cyberattacks. Unlike conventional load-balancers whichassume each data path is unique and does not affect the other,approaches may recognize that TCP and UDP share a low-level data pathand directly affect each other. Additionally, TCP provides in-orderdelivery and retransmits data (along with flow control, congestioncontrol, etc.) whereas UDP does not. This uniqueness requires additionallogic provided by the methods and systems disclosed herein that mayinclude mapping specific message types to each communication protocol,such as based at least in part on the different properties of theprotocols (e.g., expect longer jitter over TCP, expect out-of-orderdelivery on UDP). For example, the system may refrain from coding overpackets sent through TCP, since it is reliable, but may send forwarderror correction over UDP to add redundancy and save bandwidth. In someexamples, a larger ACK interval is used for ACKing TCP data.

By employing the techniques described herein, approaches distribute dataover TCP and UDP data paths to achieve optimal or near-optimalthroughput, such as in situations where a network provider's policiestreat UDP unfairly (as compared to conventional systems that simply useUDP if possible and fall back to TCP if not).

In embodiments, a method for data communication between a first node anda second node over a plurality of data paths coupling the first node andthe second node includes transmitting messages between the first nodeand the second node over the plurality of data paths includingtransmitting at least some of the messages over a first data path of theplurality of data paths using a first communication protocol, andtransmitting at least some of the messages over a second data path ofthe plurality of data paths using a second communication protocol;determining that the first data path is altering a flow of messages overthe first data path due to the messages being transmitted using thefirst communication protocol, and in response to the determining,adjusting a number of messages sent over the plurality of data pathsincluding decreasing a number of the messages transmitted over the firstdata path and increasing a number of messages transmitted over thesecond data path, wherein altering the flow of messages is performedautomatically under control of an expert system.

In embodiments, the expert system uses at least one of a rule and amodel to set a parameter of the alteration of the flow.

In embodiments, the expert system is a machine learning system thatiteratively configures at least one of a set of inputs, a set ofweights, and a set of functions based on feedback relating to at leastone of the data paths. In embodiments, the expert system takes aplurality of inputs from a data collector that accepts data about amachine operating in an industrial environment.

In embodiments, the first communication protocol is User DatagramProtocol (UDP).

In embodiments, the second communication protocol is TransmissionControl Protocol (TCP).

In embodiments, the messages are initially divided across the first datapath and the second data path using a load balancing technique.

In embodiments, the messages are initially divided across the first datapath and the second data path according to a division of the messagesacross the first data path and the second data path in one or more priordata communication connections.

In embodiments, the messages are initially divided across the first datapath and the second data path based on a probability that the first datapath will alter a flow of messages over the first data path due to themessages being transmitted using the first communication protocol. Inembodiments, the probability is determined by an expert system.

As described in US patent application 2017/0012884, entitled “Messagereordering timers,” self-organized network coding under control of anexpert system may involve methods and systems for data communicationfrom a first node to a second node over a data channel coupling thefirst node and the second node and may include receiving data messagesat the second node, the messages belonging to a set of data messagestransmitted in a sequential order from the first node, sending feedbackmessages from the second node to the first node, the feedback messagescharacterizing a delivery status of the set of data messages at thesecond node, including maintaining a set of one or more timers accordingto occurrences of a number of delivery order events, the maintainingincluding modifying a status of one or more timers of the set of timersbased on occurrences of the number of delivery order events, anddeferring sending of said feedback messages until expiry of one or moreof the set of one or more timers. The data channels may be among devicesand systems in an industrial environment, such as instrumentationsystems of industrial machines, one or more mobile data collectors(optionally coordinated in a swarm), data storage systems (includingnetwork-attached storage), servers and other information technologyelements, any of which may have or be associated with one or morenetwork nodes. The data channels may be among any such devices andsystems and devices and systems in a network of any kind (such asswitches, routers, and the like) or between those and ones located in aremote environment, such as in an enterprise's information technologysystem, in a cloud platform, or the like. Determination that that timersare required, configuration of timers, and initiation of the user oftimers may occur under control of an expert system, such as a rule-basedsystem, a model-based system, a machine learning system (including deeplearning) or a hybrid of any of those, where the expert system takesinputs relating to one or more of the types of communications occurring,the data channels, the nodes, the communication protocols used, or thelike.

The set of one or more timers may include a first timer and the firsttimer may be started upon detection of a first delivery order event, thefirst delivery order event being associated with receipt of a first datamessage associated with a first position in the sequential order priorto receipt of one or more missing messages associated with positionspreceding the first position in the sequential order. The method mayinclude sending the feedback messages indicating a successful deliveryof the set of data messages at the second node upon detection of asecond delivery order event, the second delivery order event beingassociated with receipt of the one or more missing messages prior toexpiry of the first timer. The method may include sending said feedbackmessages indicating an unsuccessful delivery of the set of data messagesat the second node upon expiry of the first timer prior to any of theone or more missing messages being received. The set of one or moretimers may include a second timer and the second timer is started upondetection of a second delivery order event, the second delivery orderevent being associated with receipt of some but not all of the missingmessages prior to expiry of the first timer. The method may includesending feedback messages indicating an unsuccessful delivery of the setof data messages at the second node upon expiry of the second timerprior to receipt of the missing messages. The method may include sendingfeedback messages indicating a successful delivery of the set of datamessages at the second node upon detection of a third delivery orderevent, the third delivery order event being associated with receipt ofthe missing messages prior to expiry of the second timer.

In another general aspect, a method for data communication from a firstnode to a second node over a data channel coupling the first node andthe second node includes receiving, at the first node, feedback messagesindicative of a delivery status of a set of data messages transmitted ina sequential order to the second node from the second node, maintaininga size of a congestion window at the first node including maintaining aset of one or more timers according to occurrences of a number offeedback events, the maintaining including modifying a status of one ormore timers of the set of timers based on occurrences of the number offeedback events, and delaying modification of the size of the congestionwindow until expiry of one or more of the set of one or more timers.

The set of one or more timers may include a first timer and the firsttimer may be started upon detection of a first feedback event, the firstfeedback event being associated with receipt of a first feedback messageindicating successful delivery of a first data message having firstposition in the sequential order prior to receipt of one or morefeedback messages indicating successful delivery of one or more otherdata messages having positions preceding the first position in thesequential order. The method may include cancelling modification of thecongestion window upon detection of a second feedback event, the secondfeedback event being associated with receipt of one or more feedbackmessages indicating successful delivery of the one or more other datamessages prior to expiry of the first timer. The method may includemodifying the congestion window upon expiry of the first timer prior toreceipt of any feedback message indicating successful delivery of theone or more other data messages.

The set of one or more timers may include a second timer and the secondtimer may be started upon detection of a third feedback event, the thirdfeedback event being associated with receipt of one or more feedbackmessages indicating successful delivery of some but not all of the oneor more other data messages prior to expiry of the first timer. Themethod may include modifying the size of the congestion window uponexpiry of the second timer prior to receipt of one or more feedbackmessages indicating successful delivery of the one or more other datamessages. The method may include cancelling modification of the size ofthe congestion window upon detection of a fourth feedback event, thefourth feedback event being associated with receipt one or more feedbackmessages indicating successful delivery of the one or more other datamessages prior to expiry of the second timer.

In another general aspect, a system for data communication between anumber of nodes over a data channel coupling the number of nodesincludes a first node of the number of nodes configured to receive, atthe first node, feedback messages indicative of a delivery status of aset of data messages transmitted in a sequential order to the secondnode from the second node, maintain a size of a congestion window at thefirst node including maintaining a set of one or more timers accordingto occurrences of a number of feedback events, the maintaining includingmodifying a status of one or more timers of the set of timers based onoccurrences of the number of feedback events, and delaying modificationof the size of the congestion window until expiry of one or more of theset of one or more timers.

In embodiments, a method for data communication from a first node to asecond node over a data channel coupling the first node and the secondnode including determining, using an expert system, based on at leastone condition of the data channel, whether one or more timers will usedto manage the data communication and, upon such determination: receivingdata messages at the second node, the messages belonging to a set ofdata messages transmitted in a sequential order from the first node;sending feedback messages from the second node to the first node, thefeedback messages characterizing a delivery status of the set of datamessages at the second node, including maintaining a set of one or moretimers according to occurrences of a plurality of delivery order events,the maintaining including modifying a status of one or more timers ofthe set of timers based on occurrences of the plurality of deliveryorder events, and deferring sending of said feedback messages untilexpiry of one or more of the set of one or more timers.

In embodiments, the expert system uses at least one of a rule and amodel to set a parameter of the determination whether to use one or moretimers.

In embodiments, the expert system is a machine learning system thatiteratively configures at least one of a set of inputs, a set ofweights, and a set of functions based on feedback relating to at leastone of the data paths. In embodiments, the expert system takes aplurality of inputs from a data collector that accepts data about amachine operating in an industrial environment.

In embodiments, the set of one or more timers includes a first timer andthe first timer is started upon detection of a first delivery orderevent, the first delivery order event being associated with receipt of afirst data message associated with a first position in the sequentialorder prior to receipt of one or more missing messages associated withpositions preceding the first position in the sequential order.

As described in US patent application 2017/0012885, entitled “Networkcommunication recoding node,” self-organized network coding undercontrol of an expert system may involve methods and systems formodifying redundancy information associated with encoded data passingfrom a first node to a second node over data paths and may includereceiving first encoded data including first redundancy information atan intermediate node from the first node via a first channel connectingthe first node and the intermediate node, the first channel having firstchannel characteristics, and transmitting second encoded data includingsecond redundancy information from the intermediate node to the secondnode via a second channel connecting the intermediate node and thesecond node, the second channel having second channel characteristics. Adegree of redundancy associated with the second redundancy informationmay be determined by modifying the first redundancy information based onone or both of the first channel characteristics and the second channelcharacteristics without decoding the first encoded data. The data pathsmay be among devices and systems in an industrial environment (eachacting as one or more nodes for sending, receiving, or transmittingdata), such as instrumentation systems of industrial machines, one ormore mobile data collectors (optionally coordinated in a swarm), datastorage systems (including network-attached storage), servers and otherinformation technology elements, any of which may have or be associatedwith one or more network nodes. The data paths may be among any suchdevices and systems and devices and systems in a network of any kind(such as switches, routers, and the like) or between those and oneslocated in a remote environment, such as in an enterprise's informationtechnology system, in a cloud platform, or the like. Modifying theredundancy information may occur by or under control of an expertsystem, such as a rule-based system, a model-based system, a machinelearning system (including deep learning) or a hybrid of any of those,where the expert system takes inputs relating to one or more of the datapaths, the nodes, the communication protocols used, or the like.Redundancy may result from (and may be identified at least in part basedon), the combination or multiplexing of data from a set of data inputs,such as described throughout this disclosure.

Modifying the first redundancy information may include adding redundancyinformation to the first redundancy information. Modifying the firstredundancy information may include removing redundancy information fromthe first redundancy information. The second redundancy information maybe further formed by modifying the first redundancy information based onfeedback from the second node indicative of successful or unsuccessfuldelivery of the encoded data to the second node. The first encoded dataand the second encoded data may be encoded, such as using a randomlinear network code or a substantially random linear network code.Modifying the first redundancy information based on one or both of thefirst channel characteristics and the second channel characteristics mayinclude modifying the first redundancy information based on one or moreof a block size, a congestion window size, and a pacing rate associatedwith the first channel characteristics and/or the second channelcharacteristics.

The method may include sending a feedback message from the intermediatenode to the first node acknowledging receipt of one or more messages atthe intermediate node. The method may include receiving a feedbackmessage from the second node at the intermediate node and, in responseto receiving the feedback message, transmitting additional redundancyinformation to the second node.

In another general aspect, a system for modifying redundancy informationassociated with encoded data passing from a first node to a second nodeover a number of data paths includes an intermediate node configured toreceive first encoded data including first redundancy information fromthe first node via a first channel connecting the first node and theintermediate node, the first channel having first channelcharacteristics and transmit second encoded data including secondredundancy information from the intermediate node to the second node viaa second channel connecting the intermediate node and the second node,the second channel having second channel characteristics. A degree ofredundancy associated with the second redundancy information isdetermined by modifying the first redundancy information based on one orboth of the first channel characteristics and the second channelcharacteristics without decoding the first encoded data.

In embodiments, a method for modifying redundancy information associatedwith encoded data passing from a first node to a second node over aplurality of data paths, the method comprising: receiving first encodeddata including first redundancy information at an intermediate node fromthe first node via a first channel connecting the first node and theintermediate node, the first channel having first channelcharacteristics; transmitting second encoded data including secondredundancy information from the intermediate node to the second node viaa second channel connecting the intermediate node and the second node,the second channel having second channel characteristics; wherein adegree of redundancy associated with the second redundancy informationis determined by modifying the first redundancy information based on oneor both of the first channel characteristics and the second channelcharacteristics without decoding the first encoded data, includingmodifying the first redundancy information based on one or more of ablock size, a congestion window size, and a pacing rate associated withthe first channel characteristics and/or the second channelcharacteristics, wherein modifying the first redundancy informationoccurs under control of an expert system.

In embodiments, the expert system uses at least one of a rule and amodel to set a parameter of the modification of the redundancyinformation.

In embodiments, the expert system is a machine learning system thatiteratively configures at least one of a set of inputs, a set ofweights, and a set of functions based on feedback relating to at leastone of the data paths. In embodiments, the expert system takes aplurality of inputs from a data collector that accepts data about amachine operating in an industrial environment.

In embodiments, modifying the first redundancy information includesadding redundancy information to the first redundancy information.

In embodiments, modifying the first redundancy information includesremoving redundancy information from the first redundancy information.

In embodiments, the second redundancy information is further formed bymodifying the first redundancy information based on feedback from thesecond node indicative of successful or unsuccessful delivery of theencoded data to the second node.

In embodiments, the first encoded data and the second encoded data areencoded using a random linear network code.

As described in US patent application 2017/0012905, entitled “Errorcorrection optimization,” self-organized network coding under control ofan expert system may involve methods and systems for data communicationbetween a first node and a second node over a data path coupling thefirst node and the second node and may include transmitting a segment ofdata from the first node to the second node over the data path as anumber of messages, the number of messages being transmitted accordingto a transmission order. A degree of redundancy associated with eachmessage of the number of messages is determined based on a position ofsaid message in the transmission order. The data paths may be amongdevices and systems in an industrial environment (each acting as one ormore nodes for sending, receiving, or transmitting data), such asinstrumentation systems of industrial machines, one or more mobile datacollectors (optionally coordinated in a swarm), data storage systems(including network-attached storage), servers and other informationtechnology elements, any of which may have or be associated with one ormore network nodes. The data paths may be among any such devices andsystems and devices and systems in a network of any kind (such asswitches, routers, and the like) or between those and ones located in aremote environment, such as in an enterprise's information technologysystem, in a cloud platform, or the like. Determining a transmissionorder may occur by or under control of an expert system, such as arule-based system, a model-based system, a machine learning system(including deep learning) or a hybrid of any of those, where the expertsystem takes inputs relating to one or more of the data paths, thenodes, the communication protocols used, or the like. Redundancy mayresult from (and may be identified at least in part based on), thecombination or multiplexing of data from a set of data inputs, such asdescribed throughout this disclosure.

The degree of redundancy associated with each message of the number ofmessages may increase as the position of the message in the transmissionorder is non-decreasing. Determining the degree of redundancy associatedwith each message of the number of messages based on the position (i) ofthe message in the transmission order is further based on one or more ofdelay requirements for an application at the second node, a round triptime associated with the data path, a smoothed loss rate (P) associatedwith the channel, a size (N) of the data associated with the number ofmessages, a number (ai) of acknowledgment messages received from thesecond node corresponding to messages from the number of messages, anumber (fi) of in-flight messages of the number of messages, and anincreasing function (g(i)) based on the index of the data associatedwith the number of messages.

The degree of redundancy associated with each message of the number ofmessages may be defined as: (N+g(i)−ai)/(1−p)−fi. g(i) may be defined asa maximum of a parameter m and N−i. g(i) may be defined as N−p(i) wherep is a polynomial, with integer rounding as needed. The method mayinclude receiving, at the first node, a feedback message from the secondnode indicating a missing message at the second node and, in response toreceiving the feedback message, sending a redundancy message to thesecond node to increase a degree of redundancy associated with themissing message. The method may include maintaining, at the first node,a queue of preemptively computed redundancy messages and, in response toreceiving the feedback message, removing some or all of the preemptivelycomputed redundancy messages from the queue and adding the redundancymessage to the queue for transmission. The redundancy message may begenerated and sent on-the-fly in response to receipt of the feedbackmessage.

The method may include maintaining, at the first node, a queue ofpreemptively computed redundancy messages for the number of messagesand, in response to receiving a feedback message indicating successfuldelivery of the number of messages, removing any preemptively computedredundancy messages associated with the number of messages from thequeue of preemptively computed redundancy messages. The degree ofredundancy associated with each of the messages may characterize aprobability of the correctability of an erasure of the message. Theprobability of correctability may depend on a comparison of between thedegree of redundancy and a loss probability.

In embodiments, method for data communication between a first node and asecond node over a data path coupling the first node and the secondnode, the method comprising: transmitting a segment of data from thefirst node to the second node over the data path as a plurality ofmessages, the plurality of messages being transmitted according to atransmission order; wherein a degree of redundancy associated with eachmessage of the plurality of messages is determined based on a positionof said message in the transmission order, wherein the transmissionorder is determined under control of an expert system.

In embodiments, the expert system uses at least one of a rule and amodel to set a parameter of the transmission order.

In embodiments, the expert system is a machine learning system thatiteratively configures at least one of a set of inputs, a set ofweights, and a set of functions based on feedback relating to at leastone of the data paths. In embodiments, the expert system takes aplurality of inputs from a data collector that accepts data about amachine operating in an industrial environment.

In embodiments, the degree of redundancy associated with each message ofthe plurality of messages increases as the position of the message inthe transmission order is non-decreasing.

In embodiments, determining the degree of redundancy associated witheach message of the plurality of messages based on the position (i) ofthe message in the transmission order is further based on one or moreof: application delay requirements; a round trip time associated withthe data path, a smoothed loss rate (P) associated with the channel, asize (N) of the data associated with the plurality of messages, a number(ai) of acknowledgment messages received from the second nodecorresponding to messages from the plurality of messages, a number (fi)of in-flight messages of the plurality of messages, and an increasingfunction (g(i)) based on the index of the data associated with theplurality of messages.

As described in U.S. patent application Ser. No. 14/935,885, entitled,“Packet Coding Based Network Communication,” self-organized networkcoding under control of an expert system may involve methods and systemsfor data communication between a first node and a second node over apath and may include estimating a rate at which loss events occur, wherea loss event is either an unsuccessful delivery of a single packet tothe second data node or an unsuccessful delivery of a plurality ofconsecutively transmitted packets to the second data node, and sendingredundancy messages at the estimated rate at which loss events occur. Anexpert system may be used to estimate the rate at which loss eventsoccur.

A method for data communication from a first node to a second node overa data channel coupling the first node and the second node such as in anindustrial environment, includes receiving messages at the first node,from the second node, including receiving messages comprising data thatdepend at least in part of characteristics of the channel coupling thefirst node and the second node, transmitting messages from the firstnode to the second node, including applying forward error correctionaccording to parameters determined from the received messages, theparameters determined from the received messages including at least twoof a block size, an interleaving factor, and a code rate. The method mayoccur under control of an expert system.

In embodiments, method for data communication from a first node in anindustrial environment to a second node over a data channel coupling thefirst node and the second node, the method comprising: receivingmessages at the first node from the second node, including receivingmessages comprising data that depend at least in part of characteristicsof the channel coupling the first node and the second node; transmittingmessages from the first node to the second node, including applyingerror correction according to parameters determined from the receivedmessages, the parameters determined from the received messages includingat least two of a block size, an interleaving factor, and a code rate,wherein applying the error correction occurs under control of an expertsystem.

In embodiments, the expert system uses at least one of a rule and amodel to set a parameter of the error correction.

In embodiments, the expert system is a machine learning system thatiteratively configures at least one of a set of inputs, a set ofweights, and a set of functions based on feedback relating to at leastone of the data paths. In embodiments, the expert system takes aplurality of inputs from a data collector that accepts data about amachine operating in an industrial environment.

While only a few embodiments of the present disclosure have been shownand described, it will be obvious to those skilled in the art that manychanges and modifications may be made thereunto without departing fromthe spirit and scope of the present disclosure as described in thefollowing claims. All patent applications and patents, both foreign anddomestic, and all other publications referenced herein are incorporatedherein in their entireties to the full extent permitted by law.

The methods and systems described herein may be deployed in part or inwhole through a machine that executes computer software, program codes,and/or instructions on a processor. The present disclosure may beimplemented as a method on the machine, as a system or apparatus as partof or in relation to the machine, or as a computer program productembodied in a computer readable medium executing on one or more of themachines. In embodiments, the processor may be part of a server, cloudserver, client, network infrastructure, mobile computing platform,stationary computing platform, or other computing platforms. A processormay be any kind of computational or processing device capable ofexecuting program instructions, codes, binary instructions and the like,including a central processing unit (CPU), a general processing unit(GPU), a logic board, a chip (e.g., a graphics chip, a video processingchip, a data compression chip, or the like), a chipset, a controller, asystem-on-chip (e.g., an RF system on chip, an AI system on chip, avideo processing system on chip, or others), an integrated circuit, anapplication specific integrated circuit (ASIC), a field programmablegate array (FPGA), an approximate computing processor, a quantumcomputing processor, a parallel computing processor, a neural networkprocessor, or other type of processor. The processor may be or mayinclude a signal processor, digital processor, data processor, embeddedprocessor, microprocessor or any variant such as a co-processor (mathco-processor, graphic co-processor, communication co-processor, videoco-processor, AI co-processor, and the like) and the like that maydirectly or indirectly facilitate execution of program code or programinstructions stored thereon. In addition, the processor may enableexecution of multiple programs, threads, and codes. The threads may beexecuted simultaneously to enhance the performance of the processor andto facilitate simultaneous operations of the application. By way ofimplementation, methods, program codes, program instructions and thelike described herein may be implemented in one or more threads. Thethread may spawn other threads that may have assigned prioritiesassociated with them; the processor may execute these threads based onpriority or any other order based on instructions provided in theprogram code. The processor, or any machine utilizing one, may includenon-transitory memory that stores methods, codes, instructions andprograms as described herein and elsewhere. The processor may access anon-transitory storage medium through an interface that may storemethods, codes, and instructions as described herein and elsewhere. Thestorage medium associated with the processor for storing methods,programs, codes, program instructions or other type of instructionscapable of being executed by the computing or processing device mayinclude but may not be limited to one or more of a CD-ROM, DVD, memory,hard disk, flash drive, RAM, ROM, cache, network-attached storage,server-based storage, and the like.

A processor may include one or more cores that may enhance speed andperformance of a multiprocessor. In embodiments, the process may be adual core processor, quad core processors, other chip-levelmultiprocessor and the like that combine two or more independent cores(sometimes called a die).

The methods and systems described herein may be deployed in part or inwhole through a machine that executes computer software on a server,client, firewall, gateway, hub, router, switch,infrastructure-as-a-service, platform-as-a-service, or other suchcomputer and/or networking hardware or system. The software may beassociated with a server that may include a file server, print server,domain server, internet server, intranet server, cloud server,infrastructure-as-a-service server, platform-as-a-service server, webserver, and other variants such as secondary server, host server,distributed server, failover server, backup server, server farm, and thelike. The server may include one or more of memories, processors,computer readable media, storage media, ports (physical and virtual),communication devices, and interfaces capable of accessing otherservers, clients, machines, and devices through a wired or a wirelessmedium, and the like. The methods, programs, or codes as describedherein and elsewhere may be executed by the server. In addition, otherdevices required for execution of methods as described in thisapplication may be considered as a part of the infrastructure associatedwith the server.

The server may provide an interface to other devices including, withoutlimitation, clients, other servers, printers, database servers, printservers, file servers, communication servers, distributed servers,social networks, and the like. Additionally, this coupling and/orconnection may facilitate remote execution of programs across thenetwork. The networking of some or all of these devices may facilitateparallel processing of a program or method at one or more locationswithout deviating from the scope of the disclosure. In addition, any ofthe devices attached to the server through an interface may include atleast one storage medium capable of storing methods, programs, codeand/or instructions. A central repository may provide programinstructions to be executed on different devices. In thisimplementation, the remote repository may act as a storage medium forprogram code, instructions, and programs.

The software program may be associated with a client that may include afile client, print client, domain client, internet client, intranetclient and other variants such as secondary client, host client,distributed client and the like. The client may include one or more ofmemories, processors, computer readable media, storage media, ports(physical and virtual), communication devices, and interfaces capable ofaccessing other clients, servers, machines, and devices through a wiredor a wireless medium, and the like. The methods, programs, or codes asdescribed herein and elsewhere may be executed by the client. Inaddition, other devices required for the execution of methods asdescribed in this application may be considered as a part of theinfrastructure associated with the client.

The client may provide an interface to other devices including, withoutlimitation, servers, other clients, printers, database servers, printservers, file servers, communication servers, distributed servers andthe like. Additionally, this coupling and/or connection may facilitateremote execution of programs across the network. The networking of someor all of these devices may facilitate parallel processing of a programor method at one or more locations without deviating from the scope ofthe disclosure. In addition, any of the devices attached to the clientthrough an interface may include at least one storage medium capable ofstoring methods, programs, applications, code and/or instructions. Acentral repository may provide program instructions to be executed ondifferent devices. In this implementation, the remote repository may actas a storage medium for program code, instructions, and programs.

The methods and systems described herein may be deployed in part or inwhole through network infrastructures. The network infrastructure mayinclude elements such as computing devices, servers, routers, hubs,firewalls, clients, personal computers, communication devices, routingdevices and other active and passive devices, modules and/or componentsas known in the art. The computing and/or non-computing device(s)associated with the network infrastructure may include, apart from othercomponents, a storage medium such as flash memory, buffer, stack, RAM,ROM and the like. The processes, methods, program codes, instructionsdescribed herein and elsewhere may be executed by one or more of thenetwork infrastructural elements. The methods and systems describedherein may be adapted for use with any kind of private, community, orhybrid cloud computing network or cloud computing environment, includingthose which involve features of software as a service (SaaS), platformas a service (PaaS), and/or infrastructure as a service (IaaS).

The methods, program codes, and instructions described herein andelsewhere may be implemented on a cellular network with multiple cells.The cellular network may either be frequency division multiple access(FDMA) network or code division multiple access (CDMA) network. Thecellular network may include mobile devices, cell sites, base stations,repeaters, antennas, towers, and the like. The cell network may be aGSM, GPRS, 3G, 4G, 5G, LTE, EVDO, mesh, or other network types.

The methods, program codes, and instructions described herein andelsewhere may be implemented on or through mobile devices. The mobiledevices may include navigation devices, cell phones, mobile phones,mobile personal digital assistants, laptops, palmtops, netbooks, pagers,electronic book readers, music players and the like. These devices mayinclude, apart from other components, a storage medium such as flashmemory, buffer, RAM, ROM and one or more computing devices. Thecomputing devices associated with mobile devices may be enabled toexecute program codes, methods, and instructions stored thereon.Alternatively, the mobile devices may be configured to executeinstructions in collaboration with other devices. The mobile devices maycommunicate with base stations interfaced with servers and configured toexecute program codes. The mobile devices may communicate on apeer-to-peer network, mesh network, or other communications network. Theprogram code may be stored on the storage medium associated with theserver and executed by a computing device embedded within the server.The base station may include a computing device and a storage medium.The storage device may store program codes and instructions executed bythe computing devices associated with the base station.

The computer software, program codes, and/or instructions may be storedand/or accessed on machine readable media that may include: computercomponents, devices, and recording media that retain digital data usedfor computing for some interval of time; semiconductor storage known asrandom access memory (RAM); mass storage typically for more permanentstorage, such as optical discs, forms of magnetic storage like harddisks, tapes, drums, cards and other types; processor registers, cachememory, volatile memory, non-volatile memory; optical storage such asCD, DVD; removable media such as flash memory (e.g., USB sticks orkeys), floppy disks, magnetic tape, paper tape, punch cards, standaloneRAM disks, Zip drives, removable mass storage, off-line, and the like;other computer memory such as dynamic memory, static memory, read/writestorage, mutable storage, read only, random access, sequential access,location addressable, file addressable, content addressable, networkattached storage, storage area network, bar codes, magnetic ink,network-attached storage, network storage, NVME-accessible storage, PCIEconnected storage, distributed storage, and the like.

The methods and systems described herein may transform physical and/orintangible items from one state to another. The methods and systemsdescribed herein may also transform data representing physical and/orintangible items from one state to another.

The elements described and depicted herein, including in flow charts andblock diagrams throughout the figures, imply logical boundaries betweenthe elements. However, according to software or hardware engineeringpractices, the depicted elements and the functions thereof may beimplemented on machines through computer executable code using aprocessor capable of executing program instructions stored thereon as amonolithic software structure, as standalone software modules, or asmodules that employ external routines, code, services, and so forth, orany combination of these, and all such implementations may be within thescope of the present disclosure. Examples of such machines may include,but may not be limited to, personal digital assistants, laptops,personal computers, mobile phones, other handheld computing devices,medical equipment, wired or wireless communication devices, transducers,chips, calculators, satellites, tablet PCs, electronic books, gadgets,electronic devices, devices, artificial intelligence, computing devices,networking equipment, servers, routers and the like. Furthermore, theelements depicted in the flow chart and block diagrams or any otherlogical component may be implemented on a machine capable of executingprogram instructions. Thus, while the foregoing drawings anddescriptions set forth functional aspects of the disclosed systems, noparticular arrangement of software for implementing these functionalaspects should be inferred from these descriptions unless explicitlystated or otherwise clear from the context. Similarly, it will beappreciated that the various steps identified and described above may bevaried, and that the order of steps may be adapted to particularapplications of the techniques disclosed herein. All such variations andmodifications are intended to fall within the scope of this disclosure.As such, the depiction and/or description of an order for various stepsshould not be understood to require a particular order of execution forthose steps, unless required by a particular application, or explicitlystated or otherwise clear from the context.

The methods and/or processes described above, and steps associatedtherewith, may be realized in hardware, software or any combination ofhardware and software suitable for a particular application. Thehardware may include a general-purpose computer and/or dedicatedcomputing device or specific computing device or particular aspect orcomponent of a specific computing device. The processes may be realizedin one or more microprocessors, microcontrollers, embeddedmicrocontrollers, programmable digital signal processors or otherprogrammable devices, along with internal and/or external memory. Theprocesses may also, or instead, be embodied in an application specificintegrated circuit, a programmable gate array, programmable array logic,or any other device or combination of devices that may be configured toprocess electronic signals. It will further be appreciated that one ormore of the processes may be realized as a computer executable codecapable of being executed on a machine-readable medium.

The computer executable code may be created using a structuredprogramming language such as C, an object oriented programming languagesuch as C++, or any other high-level or low-level programming language(including assembly languages, hardware description languages, anddatabase programming languages and technologies) that may be stored,compiled or interpreted to run on one of the above devices, as well asheterogeneous combinations of processors, processor architectures, orcombinations of different hardware and software, or any other machinecapable of executing program instructions. Computer software may employvirtualization, virtual machines, containers, dock facilities,portainers, and other capabilities.

Thus, in one aspect, methods described above and combinations thereofmay be embodied in computer executable code that, when executing on oneor more computing devices, performs the steps thereof. In anotheraspect, the methods may be embodied in systems that perform the stepsthereof and may be distributed across devices in a number of ways, orall of the functionality may be integrated into a dedicated, standalonedevice or other hardware. In another aspect, the means for performingthe steps associated with the processes described above may include anyof the hardware and/or software described above. All such permutationsand combinations are intended to fall within the scope of the presentdisclosure.

While the disclosure has been disclosed in connection with the preferredembodiments shown and described in detail, various modifications andimprovements thereon will become readily apparent to those skilled inthe art. Accordingly, the spirit and scope of the present disclosure isnot to be limited by the foregoing examples, but is to be understood inthe broadest sense allowable by law.

The use of the terms “a” and “an” and “the” and similar referents in thecontext of describing the disclosure (especially in the context of thefollowing claims) is to be construed to cover both the singular and theplural, unless otherwise indicated herein or clearly contradicted bycontext. The terms “comprising,” “with,” “including,” and “containing”are to be construed as open-ended terms (i.e., meaning “including, butnot limited to,”) unless otherwise noted. Recitations of ranges ofvalues herein are merely intended to serve as a shorthand method ofreferring individually to each separate value falling within the range,unless otherwise indicated herein, and each separate value isincorporated into the specification as if it were individually recitedherein. All methods described herein can be performed in any suitableorder unless otherwise indicated herein or otherwise clearlycontradicted by context. The use of any and all examples, or exemplarylanguage (e.g., “such as”) provided herein, is intended merely to betterilluminate the disclosure and does not pose a limitation on the scope ofthe disclosure unless otherwise claimed. The term “set” may include aset with a single member. No language in the specification should beconstrued as indicating any non-claimed element as essential to thepractice of the disclosure.

While the foregoing written description enables one skilled to make anduse what is considered presently to be the best mode thereof, thoseskilled in the art will understand and appreciate the existence ofvariations, combinations, and equivalents of the specific embodiment,method, and examples herein. The disclosure should therefore not belimited by the above described embodiment, method, and examples, but byall embodiments and methods within the scope and spirit of thedisclosure.

It is to be understood that the foregoing description is intended toillustrate and not to limit the scope of the invention, some aspects ofwhich are defined by the scope of the appended claims. Furthermore,other embodiments are within the scope of the following claims.

All documents referenced herein are hereby incorporated by reference asif fully set forth herein.

What is claimed is:
 1. A method for data communication between a firstnode and a second node over a plurality of data paths coupling the firstnode and the second node, the method comprising: transmitting messagesbetween the first node and the second node over the plurality of datapaths including transmitting at least some of the messages over a firstdata path of the plurality of data paths using a first communicationprotocol and transmitting at least some of the messages over a seconddata path of the plurality of data paths using a second communicationprotocol; determining that the first data path is altering a flow ofmessages over the first data path due to the messages being transmittedusing the first communication protocol; and in response to thedetermining, adjusting a number of messages sent over the plurality ofdata paths including: (i) decreasing a number of the messagestransmitted over the first data path and increasing a number of messagestransmitted over the second data path, or (ii) increasing the number ofthe messages transmitted over the first data path and decreasing thenumber of messages transmitted over the second data path, wherein thefirst communication protocol is User Datagram Protocol (UDP) orTransmission Control Protocol (TCP).